Federated learning is a potential ML approach that promotes cooperative learning among many distributed systems while ensuring data privacy. In this study, we present a wide review of the design and evaluation of FL, with a particular focus on data partitioning. We discuss the challenges and solutions associated with FL implementation and demonstrate the design and execution of our proposed FL architecture. The main contribution of this paper is an investigation of data partitioning in FL and its impact on system performance. Using real-world public opinion data, we evaluate our proposed FL architecture and investigate performance measures such as binary accuracy, F1 score, loss, communication overhead, and data transmission between the server and clients. The experimental results provide useful information on the efective use of FL in various contexts. We underline the distinct advantages of various data partitioning algorithms based on data distribution and privacy requirements. Our findings contribute to the creation of successful FL systems that protect privacy.
suggesting best practices for selecting and implementing data partitioning techniques in FL for public opinion survey data.
Although a variety of model performance evaluation metrics are discussed, such as communication eficiency, model performance, privacy, system performance, system and statistics heterogeneity, and motivitibility, our experimental evaluation only focuses on model performance (accuracy and F1 score) and communication overhead using public opinion data. Furthermore, this work does not address the ethical issues of FL, which would require further investigation. An additional study is required to investigate the influence of data partitioning strategies on other areas of FL system performance in order to provide an improved understanding of their efects and potential best practices.
In this context, the remainder of this paper is structured as follows: it starts with a brief review of related work and highlights the diferences and contributions of our paper in Section 2. The contribution and motivation of the research in the context of data from the public opinion survey are described in Section 2.2. The proposed design of the FL architecture is described in Section 3. Data partitioning in FL is discussed in Section 4. The performance evaluation of the FL architecture using public opinion data and presenting the metrics and techniques used for this evaluation is done in Section 5, and Section 5.1. Finally, the article concludes the work and suggests potential research directions in Section 6.
Recent years have seen a considerable increase in the level of research on FL, and many studies have been conducted on its application in various domains. In this section, a systematic literature review is applied to select and highlight relevant work on its design, application, and evaluation. The review is targeted searches in reputable databases using topic keywords to ensure completeness. Table 1 summarizes the review according to the focus area of the study, the method used to determine the main findings and limitations.
The paper [
Implementing FL can be dificult because it requires balancing the privacy and utility of local
data with the efectiveness of the ML process. Due to its distributed nature, FL encounters a
variety of challenges during training, including problems with communication, heterogeneity
of data and systems, and data privacy and security. In general, it requires careful consideration
when designing an FL system [
Table 2, describes the main issues in FL architecture due to privacy requirements and data volume, leading to limited communication in FL networks. Local updating, compression approaches, decentralized training, and importance-based updating are some of the solutions suggested by researchers. These strategies are designed to maintain the balance between effective communication, convergence, and accuracy of the model. Federated networks face the challenge of system heterogeneity, which causes participants with various communication, processing, and storage capacities. To address this issue, asynchronous communication, client participation, and fault tolerance are used. Client participation selects devices based on their resources and data quality, while fault tolerance adds algorithmic redundancy or coded computation to handle device failures.
The existence of non-IID data throughout the network causes challenges in statistical
heterogeneity in FL. To solve this, the researchers propose employing multitask learning, measuring
heterogeneity with measures such as local dissimilarity, and representing user preferences
with personalization layers. Recent research has revealed that FL may not always provide
adequate privacy guarantees during model updates and may be vulnerable to two types of
attack, including poisoning attacks and inference attacks [
Federated learning has limited use in the real world despite its benefits, such as improved model accuracy and privacy preservation. These issues can be resolved and its practical adoption improved by looking into the efects of data partitioning in FL. This article’s goal is to investigate how data partitioning techniques afect FL system performance with our following contributions: 1. An examination of data partitioning techniques in FL, focusing on how they afect system performance and communication efectiveness. 2. A novel approach to choosing and putting into practice the best data partitioning strategies for certain use cases. 3. Evaluating the efectiveness of the proposed methodology in increasing model accuracy and decreasing communication overhead using data from public opinion surveys.
The hardware and software requirements for implementing the FL architecture can vary
depending on the individual use case and the scale of the devices involved. However, in general,
clients, servers, models, and algorithms are components of the FL architecture [
Client side model evaluation and deployment
No Yes
ML frameworks that support the FL process such as TensorFlow or PyTorch. A central server software that manages the process, including model aggregation and device synchronization. which can be built with technologies like Apache Kafka, RabbitMQ, and Redis. A client-side software library that allows devices to participate in FL and communicate securely with the central server. Protocols for secure and encrypted communication to protect the privacy of data on participating devices. FL architecture design workflow for the public opinion survey example is depicted in Fig. 1, the central server distributes the initial model parameters to all clients. Clients train their local models with initial parameters and exchange the results with the central server. The central server aggregates the local models and distributes the global model to the clients.
Depending on the particular use case and privacy restrictions, many methodologies can be
utilized to evaluate and deploy the model. Clients do a local evaluation of the model and send
the results to the central server for aggregation. The performance of the global model is then
evaluated in general by the server. As an alternative, the server validation data set can be used
for evaluation. When privacy is an issue, clients can also receive the global model that has been
aggregated for local predictions. The global model, on the other hand, can be hosted by the
server and made available as a service to clients, who can then send their data for predictions.
Data privacy, resource limitations, and the complexity of the model management process are a
few examples of considerations that impact the decision to choose client-side or server-side
evaluation and deployment [
FL partitioning distributes data across multiple parties who collaborate to increase the usefulness
of their combined data. This method overcomes the limitations of domain-specific data and
makes it easier for clients with various interests to work together. Based on data flow between
parties, FL data partitioning can include transfer learning, vertical partitioning, and horizontal
partitioning (Fig. 2). It takes careful preparation to bring together the interested parties and
partition the data in a way that produces an FL environment, as proper data partitioning is
crucial for the FL process [
Horizontal FL (HFL) combines data from entities with similar features but diferent samples. In the HFL example, two research organizations (regions A and B) collect data from a public opinion survey but are only able to share limited information because of privacy concerns. The purpose is to develop an ML model that uses parameters such as age, gender, and service type to predict how satisfied clients are with government services. Each organization first trains a local model using its own data, then shares model updates with a central server to create a global model, and then deploys the aggregated model to clients.
Vertical FL (VFL) combines data from entities with the same sample IDs but distinct features. VFL allows diferent respondents to share demographic data while maintaining the privacy of survey responses. Each client trains a local model using local data and survey results and then shares model updates with the server, which constructs a global model capable of generating predictions across all attributes.
Transfer FL (TFL) involves the use of a previously trained model on a similar task to improve the performance of a new model on a new task. TFL can be applied for both VFL and HFL. TFL involves training a model on one set of data and then fine-tuning it on another. When one region has more data than the other, the model is trained on the bigger data set first and then ifne-tuned on the smaller data set.
Data in Region A
ParticipantID 001 002 003 ...
Age 35 42 55 ...
...
Data in Region B ParticipantID 001 002 003 ...
Data in Region B ParticipantID 001 002 003 ...
serviceType
A B C ... serviceType
A B C ...
Satisfactory_Level 1 4 5 ...
Satisfactory_Level 1 4 5 ...
Evaluation of FL architecture is a crucial component because it allows us to measure the
eficiency of the model and make additional improvements. Evaluation metrics, methods, and
best practices for FL architecture will be discussed in this part. The metrics shown in Table 3
are frequently used to assess the efectiveness and eficiency of the FL approach. These metrics
include communication costs, model performance, system scalability and performance, attack
rates, computation and energy costs, convergence rates, statistical and system heterogeneity,
client motivation, and data and device security [
The performance of HFL, VFL, and TFL was compared using data collected by the Asia Foundation. It was a public opinion survey to obtain civilian thoughts and impressions on a variety of Afghanistan-related issues. The data set includes survey questions and responses related to security, governance, and country development.
Due to the FL nature, clients send their local model updates to the server, which aggregate these updates to improve the global model. Communication is a substantial bottleneck in the FL process, especially if the network bandwidth is limited or a large number of clients are participating. In our experiments, the quantization technique was investigated as a substantial solution to this problem for three FL architectures (HFL, VFL, and TFL). The findings showed in Fig. 3 that the transmission overhead without quantization was HFL:14.44 MB, VFL:0.12 MB, TFL:17.32 MB, while, the communication overhead for the three techniques decreased significantly after applying quantization to model updates as HFL:7.22, VFL:0.06, TFL:8.70.
These findings indicate the eficiency of quantification in reducing communication overhead in FL systems. Reduced overhead can result in faster convergence, improved scalability, and lower communication costs. However, it is critical to assess the impact of quantization on the accuracy and loss metrics of the model. In this research, we also conducted tests for binary classification models in horizontal, vertical, and transfer FL setups. The results showed that the use of quantization maintained acceptable levels of accuracy and loss, making it a feasible solution to reduce communication overhead in FL systems.
Table 4, summarizes the experimental methodology and analysis performed in our study. It includes key elements such as the description of the data set, the comparison of diferent data partitioning methods, the specific federated learning model used, the hyperparameters chosen for training, the evaluation metrics used, and an outline of the experimental procedure, details about the analysis process, and the source of the results shown in Fig. 3, Fig. 4, Fig. 5, and Fig. 6. Mentioned and presented figures represent the author’s contribution in this research.
Accuracy
F1 Score
Loss
Accuracy
F1 Score
Loss 0.0 0 20 4R0ounds 60 80
4R0ounds 20 60 80
The performance of three diferent FL approaches for HFL, VFL, and TFL is studied. The performance of each approach is evaluated using three metrics: test loss, accuracy, and F1 score. The findings of the HFL experiment are indicated in Fig. 4. theTtheestinacitciualralcoyssiso0f.5t3h,eanHdFtLheteFs1t sicsor0e.6i9s, 0.90 Accuracy F1 Score Loss 0.54. Model performance improved consis- 0.88 tently throughout 90 rounds, with the test re loss dropped to 0.32, the accuracy increased coS10.86 to 0.85, and the F1 score increased to 0.83. As /Fy shown in Fig. 5, the first test loss for verti- rccau0.84 cal FL is 0.66, the accuracy is 0.55, and the cA F1 score is 0.0. Over 90 rounds, the model 0.82 improved in all metrics, with the test loss decreased to 0.30, the accuracy increased to 0.85, 0.80 0 20 40 60 80 and the F1 score increased to 0.84. Rounds i0n.8Ft4hi6ne1a,ilnlayni,tdifaotlrhdTeaFFtLa1issnectoFirigse. 0i6s.,30t2h.,8e2th9ine2i.taiTcahcluetermsatcolyodseissl Figure 6: tThFoLr’MscoodnetlriPbeurtfioornm. ance. Source: auimproved during 90 rounds, with the test loss dropped to 0.30, and the accuracy and F1 score slightly increased to 0.8480, 0.8329 respectively.
In summary, during 90 rounds, the three FL approaches showed a continuous improvement in performance in all evaluation metrics. The findings show that FL can be eficiently applied to a variety of situations, each strategy providing unique benefits based on specific data distribution and privacy needs.
The paper provides a thorough investigation of the architecture of FL, with a particular emphasis on data partitioning. The importance of FL has been emphasized and the problems and limitations of existing FL techniques have been studied. The design concepts and factors necessary to establish an FL architecture have also been investigated. FL architectures have been evaluated using metrics and approaches related to data partitioning strategies. The implementation and evaluation of the FL architecture was carried out using various data partitioning architectures and the results were thoroughly explained. By evaluating the FL system using new measures, future development of more eficient, efective, and privacy-preserving FL systems can be helped. These measures should address statistical and system heterogeneity, system performance, client motivation, system scalability, and data privacy, in particular. Taking these factors into account, we can improve our understanding of FL, leading to the development of more eficient, efective, and secure FL learning systems.
This publication has been written thanks to the support of the Operational Programme Integrated Infrastructure for the project: International Center of Excellence for Research on Intelligent and Secure Information and Communication Technologies and Systems – Phase II (ITMS code: 313021W404), co-funded by the European Regional Development Fund. It is also supported by the Operational Program Integrated Infrastructure for the project: National infrastructure for supporting technology transfer in Slovakia II – NITT SK II, co-funded by the European Regional Development Fund, and the AI4EOSC project under grant number 101058593.