The rapid growth of distributed data across edge devices has prompted the development of decentralized machine learning techniques, such as Federated Learning (FL), to address privacy and data transfer concerns. Only a few recent works have focused on unsupervised FL approaches compared to their supervised counterparts, with the consequence that many aspects of these solutions, e.g., the communication cost, have not been thoroughly investigated. In this paper, we analyse the communication cost associated with unsupervised federated anomaly detection, focusing on a proposed method where clients are grouped into communities based on inlier patterns and subsequently train autoencoder models in a federated fashion. Our analysis quantifies the communication overhead introduced by the federated learning process and compares it to traditional centralized approaches for anomaly detection. We also explore potential trade-ofs between communication cost, privacy, and model performance. Our findings reveal that the unsupervised federated approach can achieve a significant reduction in communication cost (up to 83.33%) with comparable performance, by selecting better-suited models. Furthermore, the adjustments we implement render the methodology independent of dataset size, ofering notable privacy benefits and competitive accuracy performance, making it highly efective in industrial scenarios with large local datasets and a moderate number of clients.
The capacity to obtain valuable insights from Big Data through AI techniques is a key component for the advancement of Industry 4.0 [1]. In this scenario, robots near production lines gather, process, and utilize data generated during their operations. These robots have built-in computing and communication capabilities for AI tasks such as identifying potential failures or product defects and coordinating with one another.
The prevalent AI paradigm is centralized, where data gathered by edge devices are transmitted to the cloud for AI model training. However, due to the explosive growth of data generated at the edge and heightened concerns about data privacy and ownership [2], there is a shift towards relocating AI processes to the edge, transitioning the paradigm towards a more distributed or decentralized approach.
Federated learning (FL)[3] has arisen as an attractive method for training machine learning models without the need to share raw data among diferent clients. This decentralized framework maintains data privacy, renEvelop-O
Federated Learning (FL) is a distributed learning framement on edge devices, and has quickly advanced since (L. Valerio); 0000-0002-1694-612X (A. Passarella)
© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License work that optimizes computing power and data manageits introduction [3]. FL provides a modern and more ef- pairs of clients exchange models to classify their local fective evolution to traditional distributed paradigms [6, data. If nodes have similar inlier/outlier ratios using 7, 8, 9, 10] and has been deployed in various use-cases by each other’s models, they share the same inlier class and major service providers [11, 4, 12]. should be in the same group.
Unsupervised FL has been relatively less studied, with An undirected graph is generated using candidate a few notable works in the field. For instance, Van [ 13] groups from each client, and a community detection alifrst introduced unsupervised representation learning in gorithm is applied to identify groups of nodes for the a federated setting, but without considering typical dis- upcoming standard FL step. tributed setting issues. Zhang et al.[14] addressed the inconsistency of representation spaces and misalignment of 3.1.2. Federated Learning Outlier Detection representations in unsupervised FL. Tzinis et al.[15] proposed an unsupervised FL approach for speech enhance- After the first phase, groups (or communities) 0, … , ment and separation, utilizing a small portion of super- are formed. For each group, federated learning is initiated vised data to boost the main unsupervised task. In [16], using autoencoders as models. Autoencoders are suitable Luo and colleagues presented a collaborative system of because they naturally fit the FL framework and can autoencoders for distributed anomaly detection, but with efectively be used in AD tasks. The Federated Averaging local models used for inference only. Mothukuri et al. [17] (FedAvg) protocol is used for FL [3]. At the end of each proposed a FL-based anomaly detection approach for in- communication round, the trained autoencoder is shared trusion identification and classification in IoT networks. among the clients of the same group. Lubana et al.[18] proposed Orchestra, a globally consistent clustering technique for unsupervised FL that guar- 3.2. Cost of Decentralized Anomaly antees good generalization performance. Han et al. [19] Detection introduced FedX, an unsupervised FL framework that employs a two-sided knowledge distillation with contrastive 3.2.1. Phase I learning, improving performance on five unsupervised algorithms.
We use the distributed anomaly detection framework presented in [5] as our reference, revisiting its methodology and analyzing communication costs.
We study a distributed learning system with clients and data distributions , where || ≤ | | . Each client obtains a fraction of its samples from ∈ ⧵ , with ≠ , and the remaining (100 − )% from ∈ . Typically, ∈ [5%, 15%] is considered realistic and is commonly used in anomaly detection contexts [20].
The methodology’s goal is to create consistent groups of clients and perform standard federated learning within those groups.
In the subsequent sections, we revisit the two-phase structure that constitutes the entire process, (for more details, please refer to the reference work [5]). 3.1.1. Preprocessing Phase: Group Identification This phase aims to enable clients to join a group with the same (or similar) majority class . Clients train a lightweight anomaly detection model on local data, and Firstly, each client trains a model on its local data and exchanges it with the other clients. Given | | = clients and letting be the size of a single local model, the total communication cost for exchanging models between all pairs of clients can be estimated as ( − 1) × . Clients then share pairwise association information, adding a cost of ( − 1) × , with = 1 bit.
Clients share candidate groups with a single client, who builds the graph, runs the community detection algorithm, and sends back the community information.
A predefined policy selects the client for this task, like the one with the lowest ID.
Assuming the average candidate group size is and each client ID size is , the total communication cost for sharing candidate groups to the selected client can be estimated as ( − 1) × × . can vary depending on the performance of the first step. In the worst case, is equal to the number of clients, i.e., | | = . In the ideal case, is equal to the average size of the real groups, where each group is a set of nodes sharing the same data distribution. Assuming that the average number of nodes in each group is ≥ 1 , it holds that in the ideal case = = /|| .
In our experiments, we observed that typically aligns with this ideal condition, demonstrating our methodology’s efectiveness in grouping clients with similar data distributions. After the community graph has been computed, an additional cost of ( − 1) × × is required for sending community information back to munity) 0, … , starts a federated learning instance using a corresponding model 0, … , (they all have the same architecture). The communication cost analysis in this phase involves local model updates, model aggregation, and global model update distribution. For simplicity, we firstly consider a single group and an external aggregator. Each client trains their model on local data and computes updates. The size of these updates depends on the model architecture, and we denote it as . Clients share local updates with the aggregator, which combines these updates using the Federated Averaging algorithm.
The communication cost for sending local model updates is | | × for each group , where | | is the number of clients in group . Afterward, the aggregator distributes the global model update to all clients in the group, with a communication cost of | | × .
The total communication cost for the second phase is tion rounds in each FL instance and assuming there are groups, the formula for the total communication cost in the second phase is: 2 = ∑ 2| | =0
Considering that the sum over | | counts all the clients in the system, and that selecting a client within the group as the aggregator slightly improves the communication cost (i.e., that client does not need to send its update), we can rewrite the total communication cost formula for the second phase as follows: 2 = 2 ( − )
Where is the number of clients. Asymptotically, the communication cost in the second phase is linear with respect to the number of clients and the size of the local model updates.
(2) (3) the clients. Therefore, the overall communication cost for the first phase can be summarized as: 1 = ( − 1) + ( − 1) + 2( − 1)
||
(
The first phase’s communication cost is dominated by the quadratic term ( − 1) . As the number of clients grows, this impacts the cost significantly. The model size, , also becomes a dominant component in the cost compared to other information types. Thus, the first phase’s communication cost can be expressed as ( emphasizing the model size’s importance in the overall 2 ), cost. 3.2.2. Phase II In the second phase of the methodology, each group (com- in terms of privacy of data, but also in terms of commuthe sum of the costs for all groups. Given communica- ideal partitioning we aim to find consists of = || = 10 communication cost of ∑ the size of a single data point.
=1 | On the other hand, in a centralized solution, all clients send their local datasets to a central server for processing, and the server trains a global model on the entire dataset . The communication cost in the centralized approach is linearly dependent on the dataset size. Clients send their entire local datasets, , to the server, resulting in a
| × , where represents
Recalling that the cost of the proposed decentralized solution is quadratic in the number of clients and linear in the local model size, the choice between the proposed methodology and a centralized solution depends on the specific requirements and priorities of a given application. For instance, when dealing with large datasets, the proposed methodology is likely more eficient not only nication. When, instead, data are very small in size, a decentralised solution may generate higher trafic than a centralised one. In such cases, it is crucial to utilize relatively light models in the decentralized approach to further optimize communication costs and ensure its effectiveness in comparison to the centralized solution.
We evaluate the proposed methodology using the fashionMNIST [21] dataset and focus on the communication aspect. We follow the experimental setup in [5], with minor variations.
The fashion-MNIST dataset has ten classes (|| = 10 ). We ensure that clients have numerically balanced and disjoint datasets. We set = 9 , representing the number of clients within the same data distribution (class). The groups with clients each. We only consider the case of = 9 for this evaluation.
The models used here and in the reference work are presented in Table 1. All the number of parameters reported for the models refer to the input of MNIST-like datasets, i.e., 784 dimensions.
In contrast to our prior work [5], in the first phase, we replace OC-SVM [22] with a small convolutional autoencoder to break the dependence on the data size (thus reducing the communication cost), while maintaining satisfactory performance (in terms of detection accuracy). Notably, the OC-SVM model does not have a fixed number of parameters, as it depends on the support vectors found during the training process; hence, the number of parameters is proportional to the local dataset . In the second phase, we transition from a flat autoencoder to a more extensive convolutional autoencoder, which enables better representation of the underlying structure
In this section, we present the experimental results of the group detection process for the Fashion-MNIST dataset. tional models; similar findings regarding communities were observed in our prior work[5], albeit at a higher communication cost. ifne
Let , be the j-th client with majority class ; we deas the ideal set of clients having the same majority class , e.g., 0 = 0,0, … 0,−1 . In table 2, we observe that most of the communities found for fashion-MNIST consist of clients with the same majority class, such as 0 with 6, 1 with 8, and so on. An exception is given by 6, which is formed by clients with majority classes in these majority classes. 4.2. Federated Outlier Detection ology and test protocol as in our prior work [5]. For instance, we take two baselines as reference: (i) local, where clients only train on local data; and (ii) ideal, in which a client , uses the model trained through federated learning on the set of clients , i.e., the set of the clients sharing the same majority class. The test samples for each client are randomly sampled from the The results presented in Table 3 show the average AUC-ROC scores across clients having the same inlier class. Our proposed configuration outperforms the local baseline in all inlier classes, indicating that the federated outlier detection approach efectively leverages global thermore, our results are consistently close to the performance of the ideal baseline, demonstrating the potential of the methodology to achieve near-optimal results.
This positive performance trend is also evident for clients that are trained within the same federation. For instance the large federation of clients with classes 0, 2, 3, and 4 demonstrates competitive performance compared to the ideal baseline, where a federated instance is run for each ideal group.
In our prior work [5], mean AUC-ROC values of 0.714, require higher communication overhead compared to the models we propose. Here we achieves comparable results with lighter models, as demonstrated by the mean AUCROC of 0.649, 0.728, and 0.740 in Table 3. This highlights the eficiency and efectiveness of proper model selection, as it reduces communication overhead while maintaining competitive performance.
parameters multiplied by 4 bytes.
To calculate the actual communication costs for the two
phases of the methodology, we can use Eq.(
We will consider each parameter to be represented using 32-bit floating-point numbers. Therefore, the size of a model ( or ) is determined by the number of its
In our experiment with 90 clients, each client ID (i.e., mately 0.522 MB.
Regarding OCSVM, the size of the model is linearly dependent on the size of the client’s local dataset . On average, in our test, a trained OCSVM model has a size of 0.3 , where is approximately 1/90 of the fashionMNIST train set. The fashion-MNIST train set is about 47.04 MB in total, resulting in an average of approxi 2, 0, 3, and 4. This indicates that there is a higher de- 0.761, and 0.772 were obtained using the OCSVM and the gree of similarity between the clients’ data distributions flat autoencoder model shown in Table 1. These models We compare our implementation with the same method- ) can be represented using a 2-byte unsigned integer.
In this paper, we analyzed a decentralized and unsupervised anomaly detection solution [5] from the perspective of communication costs. We derived closed formulas to estimate the costs and discussed model choices for eficient anomaly detection in terms of accuracy and communication cost.
Our results demonstrated that the proposed methodology could detect similar communities to the reference work while significantly reducing communication costs by approximately 83.33%, with comparable performance. Importantly, the communication cost reduction is independent of the dataset size, unlike in the reference work or centralized data approaches.
The methodology is well-suited for distributed industrial applications with a moderate number of clients, as its communication cost is quadratic with respect to the number of clients (( 2 )). By using smaller models relative to the dataset size, this approach is more convenient than centralizing the data and is highly efective in various contexts, particularly when centralized data storage is impractical or restricted by privacy constraints.
In future work, we aim to explore the potential of other models and techniques to further optimize communication costs (e.g., remove the quadratic complexity with respect to the number of clients) while maintaining high accuracy in decentralized anomaly detection. We also plan to investigate the applicability of our methodology to a broader range of industrial scenarios and datasets, such as video or IoT sensor data. Additionally, the development of adaptive communication strategies to dynamically adjust the number of communication rounds or model selection based on the current network conditions or dataset properties could be an interesting avenue for further research.
This work has been partly funded under the H2020 MARVEL (grant 957337) and CHIST-ERA SAI (grant CHISTERA-19-XAI-010) projects. The work of A. Passarella has been partly funded by PNRR - M4C2 - Investimento 1.3, Partenariato Esteso PE00000013 - ”FAIR - Future Artificial Intelligence Research” - Spoke 1 ”Human-centered AI”, funded by the European Commission under the NextGeneration EU programme. and opportunities in the hype of industry 4.0, in: 2017 IEEE International Conference on Communications (ICC), IEEE, 2017, pp. 1–6. Z. Lin,
When machine learning meets privacy: A survey and outlook, ACM Computing Surveys (CSUR) 54 (2021) 1–36.