Federated Learning is a well-known learning paradigm that allows the distributed training of machine learning models. Federated Learning keeps data in the source devices and communicates only the model's coeficients to a centralized server. This paper studies the decentralized flavor of Federated Learning. A peer-to-peer network replaces the centralized server, and nodes exchange model's coeficients directly. In particular, we look for empirical evidence on the efect of diferent network topologies and communication parameters on the convergence in the training of distributed models. Our observations suggest that small-world networks converge faster for small amounts of nodes, while xx are more suitable for larger setups.
Federated Learning (FL) [
FL ofers a series of out-of-the-box benefits that it makes it attractive to academia and industry, including: (i) Models constantly improve using nodes data with no need to aggregate data in a centralized server; (ii) Source data are kept close to the source enabling privacy-aware learning; (iii) FL also fits situations when the network is limited in bandwidth and latency.
Recently, research communities devoted much efort to decentralizing the learning process.
Nevertheless, the study of the benefits of fully decentralized against centralized solutions is still
a challenge and is of particular interest [
The content of this discussion paper is based on another manuscript [
Hegedus et al.[
Federated Learning comes in many flavors (see [
More formally, each node has a local dataset , such that the whole dataset = {1 ∪ · · · ∪ } and that ∩ ′ = ∅ for ̸= ′. The training process of horizontal FL is agnostic with respect to the specific learning model used. Commonly used models in FL exploit a gradientdescent strategy to minimize the value of a loss function (· ) (a function that gives a measure of the error), defined on the parameters vector , i.e. ().
Procedurally, the training phase of the model is usually composed of the following steps
[
By comparison with the centralized FL, the general objective of decentralised learning is to minimize the average of the loss functions for all the nodes in the system. Indeed, the core diference between centralized and decentralized federated learning is, as the name implies, that the latter does not require a centralized server to work. Instead, the training phase and model distribution are performed only with peer-to-peer communications between nodes.
We model the communication network with a undirected graph of nodes. We define the neighborhood of a user , denoted as , as the set of nodes that have an edge in the network going from to . We also consider the edge to be bi-directional, i.e. if ∈ then ∈ . Nodes communicate in rounds, with each rounds having the same length for each node, Δ. In our model does not change over rounds and remains static.
The training procedure is the following: (i) Each node initially trains its model on the local
and obtain ; (ii) for each Δ, the is communicated to a subset of , according to
the available bandwidth. Here, we assume that each neighbour also communicates back its
own . (iii) upon the reception of a nodes update their model by means of an aggregation
Size
Number of features
Number of labels
Label distribution
function (· ). The aggregation function is a crucial element when considering the quality of a
model. Since our focus is not on the quality, here we consider a simple aggregation function
that performs an average of the coeficients [
All the experiments have been run in Python 3.8 on a single workstation machine. The Scikitlearn1 library was used for the classifiers. The networks were generated with the NetworkX 2 library.
Datasets. For the experiments, we have considered two datasets prepared for multi-label classification tasks. The properties of the datasets are listed in Table 1. The first dataset is the Pendigits3, in which each sample contains features taken from handwritten digits from 250 diferent subjects. The task is to classify each image with the correct digits. In the second dataset, HAR4, four human activities (inactive, active, driving, walking) were recorded with a mobile app and put in relation with several sensors measurements in the phone. The task is to classify the activities using data from the accelerometer, gyroscope, magnetometer, and GPS. For most of the experiments, we randomly selected 10K samples from the original dataset (which consists of around 500K samples) to be similar in size to the Pendigts dataset.
The 10% part of each dataset is reserved for testing, and the remainder 90% is distributed to nodes for training. The training data has been divided among clients proportionally, i.e., each client roughly receives the same amount of samples. This means that when testing larger networks, the number of samples is lower per node: this is the cause of lower general quality in the prediction power of larger networks. Data does not change during the simulation. Models. We used two machine learning classifiers in our experiments. The first one is a stochastic gradient descent (SGD) model that trains linear Support Vector Machine using the hinge loss function. The second one is a Logistic Regressor classifier. Since logistic regressors are used for binary classification problems, we use the one-vs-rest (OvR) strategy for the multi-class classification. The OvR subdivides the original multi-classification problem into multiple binary sub-problems (one for each class) and trains a model for each sub-problem. The model that
2https://networkx.org/ 3http://archive.ics.uci.edu/ml/datasets/pen-based+recognition+of+handwritten+digits 4https://lbd.udc.es/research/real-life-HAR-dataset/ obtains the best results is used for the prediction.
Networks. We use three diferent types of networks derived from graph theory to build the overlay between nodes. For each overlay, we assume a bidirectional communication channel that corresponds to an undirected graph. Nodes have no self-edges or double edges. All networks are also fully connected.
• Regular random graph. Every node has the same degree. Neighbors are chosen randomly. • Small-world graph. It is built with the Watts–Strogatz technique5. It has short average path lengths, i.e., the distance between two randomly chosen nodes is proportional to the logarithm of the graph’s size. Some social networks and biological networks can be modeled with a small-world graph. • Scale-free graph. It is built with the Barabási–Albert preferential attachment technique.
These graphs have nodes, called hubs, with a disproportionately large degree compared with other nodes. Many networks can be models as a scale-free graph, including the Internet and many social networks.
These networks are characterized by two parameters. The first one is the size of the network = {32, 64, 128, 256, 512}. The second parameter = {5, 10, 15, 20} controls how nodes connect to each other. The semantic of this parameter is diferent with respect to the network considered. In the random graph, it defines the degree of the nodes; in the small-world graph, it defines the number of nearest neighbors that each node connects to in the ring topology; in the scale-free graph, it defines the number of edges to attach from a new node to existing nodes. Communication. The communication among nodes is divided into synchronous rounds, or iterations, in which each node has a chance to communicate and exchange the model coeficients with another node. Each node can communicate only with its neighbors on the network. An iteration terminates when all nodes have had the chance to communicate. Communication between nodes is afected in two ways. First, we simulate connections between nodes not working properly by dropping some communication with various percentages, = {0, 0.1, 0.2, 0.3}. When a communication is dropped the corresponding model information is lost. Second, we define the maximum number of nodes communications per rounds, = {1, 2, 4, 8}. For example, when = 1, each node can communicate with only one neighbor at each iteration. The selection of neighbors is made accordion to a round-robin algorithm. When < , node communicates with all its neighbours at each iteration.
The system is evaluated by considering how fast each experiment converges to a value for the model accuracy, i.e., the fraction of correctly classified items. We have also measured other quality metrics such as precision, zero-one-loss, and f1 score, and we observed that the results are comparable with the accuracy. The system accuracy is the average of the accuracy values
of all nodes in the network. It is computed at the end of every iteration. The convergence is measured by counting the diference between consecutive measurements of the system accuracy. We consider the system to have converged if, for three consecutive times, the accuracy is not lower than the previous value and the diference is less than 0.001.
Figure 1 shows the influence of the degree parameter on the convergence and the overall accuracy of the decentralized learning. On the random graph, the degree has a high impact in terms of overall classification accuracy. In particular, a 5 degrees network converges to an accuracy slightly above 86%, whereas the 20 degrees network converges to 88%. The speed convergence is also afected, with the 10 degrees network converging earliest at the 13th iteration. The small-world network obtains similar values for the final accuracy. There is a marginal diference between the various degree values, having a similar trend in all cases. Similar to the small-world network, the scale-free network shows marginal diferences in all cases. We can notice significant "steps" in the increment of accuracy due to those iterations in which high-degree nodes are updated with better models.
Figure 2 shows how the accuracy varies over time with the size of the network . It is worth noticing that the lower maximum accuracy obtained by larger networks is because the amount of data per node is much lower, skewing the global accuracy drastically. However, we can notice that smaller networks reach convergence faster than larger ones, but no significant diferences can be seen for diferent kind of networks.
Although decentralized federated learning is gaining momentum, only a few works analyze its behavior related to the network topology. This paper evaluated the impact of diferent network characteristics on the convergence speed of a decentralized federated learning system. In particular, we have empirically evaluated how sensitive is the training process for the various network characteristics and parameters.
Our results suggest that clustered networks such as scale-free and small-world look more suitable to support decentralized training than other kind of networks. In particular, small-world networks seem to converge faster for a small setup when the amount of training sample per node is relatively low. By comparison, scale-free networks obtained better results in the largescale test. That could indicate that having a hierarchical organization of the network in which hubs (or super peers) can collect the aggregated models and redistribute them could increase the convergence speed. Naturally, this is a trade-of, as hierarchical systems also increase the load on specific nodes and are sensitive to the single point of failure (if a super-peer becomes unavailable, much valuable information is lost). We reserve to study these trade-ofs in future work, also considering dynamic networks, i.e., that changes during the training phase.
This work has been partially supported by the European Union’s Horizon 2020 Research and Innovation program, under the project TEACHING (Grant agreement ID: 871385).