Classifying computer network devices by their behaviour is crucial for keeping track of today's dynamically changing network environments. In the field, it is mainly done either manually or using heuristics. The commonly used solutions completely overlook the computer network's inherent graph structure. In this paper, we show that leveraging the graph structure using graph convolutional neural networks is worth the added computational burden. Our evaluation on real-world networks with user-defined classes includes three baseline models and a graph neural network. The experimental results highlight the models' proficiency in learning diverse device classes, with the graph-based models exhibiting superior performance. We also show that the graph-based models struggle to adapt to the ever-changing structure of the network and measure the cost of their retraining.
For this purpose, we designed features that capture
important device information without building a graph
Properly managing and monitoring large private net- and features computed from the graph structure. We
works is necessary for companies to avoid network fail- train and test various models on real-world, large-scale
ures that could cause business disruptions. An essential networks spanning multiple industries and numerous
part of this process is knowing the role and importance of distinct device types. Consequently, the model evaluation
each device in the network. Devices crucial for business provides valid information on the usability of the models
operations must be diferentiated from the less important in the real world.
ones. The impact of losing connection to a smart light
bulb difers vastly from losing a production server.
Nowadays, manual device classification is still common. But 2. Prior art
it is time consuming and requires extensive domain and
local network environment knowledge. This makes it Behavioural device classification based on network trafic
impractical for large-scale networks. Automated or semi- can be based solely on static information about the device.
automated heuristic solutions exist, but the dynamic na- This group of methods is in this paper represented by
ture of networks, where devices are constantly added the three standard ML models: Support Vector Machine
and removed, further complicates the labelling process. (SVM) [
Our approach to the device identification problem is creating node embeddings for classification. The
Matrixas a semi-supervised classification. We assume that by factorisation leverages the eigendecomposition of a
proxleveraging the graph structure of the computer network, imity matrix as the embedding for each node [
There is no universal template for categorising devices
within private networks. The type of devices in networks
varies greatly alongside the industry the company is
operating in. Consequently, the companies usually design
introduced in GraphSage [
The embedding of each node contains aggregated in- To leverage the structure of the computer network, we
formation from its -hop neighbourhood, and due to consider devices to be nodes, and communication
beits locality, it scales much better than the previous ap- tween two devices indicates an edge. To prevent the
proaches. The GraphSage has been surpassed on common graph from expanding to unreasonable sizes by
reprebenchmarks by other architectures, such as GAT [
engineering The novel positional features are constructed as
follows: We choose nodes with the highest PageRank
In this paper, we work with the network communication centrality [16] as central nodes for the network. These
logs collected by a lightweight application (collector) in- central nodes then serve as anchors in the graph. Each
stalled on the endpoint device. The data format is similar node’s global position in the graph is then encoded into
to the NetFlow [
For feature selection and engineering, we used con- distribution are displayed. The steep improvement in nection logs from 8 real-world networks of various sizes the results occurs at 20 central nodes, corresponding to (from thousands to hundreds of thousands of devices), the number of important nodes in the respective graph diferent industries (logistics, healthcare, finance, . . . ) and with respect to the PageRank centrality (according to a labelled by diferent logic (by location, function, or com- deeper graph analysis). Since other networks may have bination). a larger graph with more central nodes, e.g. Company C has 30-40 central nodes, we have decided to set the 3.1. Device features number of considered central nodes to 50 throughout the experiments. Further increasing the number of central nodes did not have any impact on the classification accuracy.
Each device in the network can be described by a set of features engineered from a device’s communication logs without building the local network’s graph. In total, we use 122 features, each computed from the device communication spanning a single day. The comprehensive list of features can be found in Appendix A. We divide these features into four categories - port, time, networking and hash. The port number can indicate what services are running on the device and what services it label
City2 A City3 n City4 y ap City5 o City6 m C City7
City8 B Domain Controller n Protect y a p Protect - IT om Server C Loc. A - servers
Loc. A - workers Loc. A Building Services Loc. A IS
Loc. A Lab C Loc. B - servers y Loc. B - workers a Loc. B App Packaging n p m Loc. B Cardiology EEG o C Loc. B Cardiology PACS
Loc. B Medical Device Loc. B Philips Software Loc. B Radiology Loc. C General Loc. C General Srvs labelled devices 0.27 0.52 0.67 the device classes themselves, resulting in diferent types of grouping. One company may want to group the devices by their geographic location, while another one by the function of the device in the network. From the eight networks used for feature selection, we chose the three most diverse to use in the final experiments. The labels for our experiments were provided by the respective companies themselves. Table 2 shows the device counts and labels for three networks.
The ratios of labelled devices within the networks further support the semi-supervised learning scenario. The models can not only extrapolate learned knowledge to unlabelled devices, but the GNN model leverages the information from unlabelled devices in its aggregation scheme while classifying the labelled devices. The number of devices within each class indicates severe class imbalance for Company B and Company C. During the evaluation, special countermeasures had to be taken for both companies. weights [17] were applied for training SVM, Random Forest, and GNN. AdaBoost performed well without class weighting. For Company C, the majority class covers ∼ 90% devices. Therefore, it was subsampled before the
The models were set up by hyperparameter grid search We designed two experiments to evaluate the usability of with 4-fold cross-validation, based on the study [18]. Reproposed solutions in the real world. The first experiment garding GNN architecture, initial experiments revealed confirms whether models can learn the diverse behaviour that one hidden layer aggregating a 1-hop neighbourof network devices when training on each day separately. hood is suficient. Deeper architectures resulted in worse The second experiment tries to find out how long models performance. Additional GNNs parameters, such as the maintain good classification results before they need to number of heads in the attention mechanism or dimenbe retrained. sion of the final embedding, were optimised using the Tune framework [19], utilising the Adam [20] optimiser.
In the experiments, we worked with a dataset collected over 14 days on three private networks of various sizes and structures, see Table 2. Both device and graph features were calculated for each day.
One of the substantial challenges in the training process was the class imbalance. In Company B, class
The first experiment was designed to evaluate models’ performance over a working week of telemetry from two perspectives. Firstly, how well can the models learn the diverse classes. Secondly, to assess whether the models’ training is stable throughout the week. Stability is an essential metric for potential real-world use. To validate training stability, models were trained and evaluated on ten train/test splits each day, as shown in Figure 2. Specifically, for each train/test split and day, each model was trained and evaluated on data from that particular day. We provide experimental analysis for each company separately to discuss the caveats of diferent networks. 4.2.1. Company A 4.2.2. Company B Due to severe class imbalance in Company B, we present the macro average recall and precision, rather than accuracy in Figure 4.
The class imbalance, specifically the low number of devices in certain classes of the testing dataset, causes substantial fluctuations in macro-averaged metrics, resulting in unstable reported scores and wide confidence intervals. To illustrate the problem, an example of classification results of the GNN model is shown in Table 3. In this example, the misclassification of one domain controller and one device from Protect-IT class would cause the reported macro average for recall to fall by more than 10%.
Protect Protect-IT
Server macro avg accuracy 4.2.3. Company C The classification results for Company C are also afected by class imbalance. The classification performance depicted in Figure 5 shows that the GNN model with AdaBoost is consistently more precise than other baseline models.
Further study of the confusion matrix of the GNN model, in Figure 6, shows two extremes, either the model can distinguish the class well or almost not at all. The model usually classified all devices from classes Loc. A Lab, Loc. A IS, and Loc. A Building Services entirely into the Loc. A− workers class. This is not surprising as all three classes fall within the Loc. A− workers definition (workstations, not servers, in the same location). Similar results from other models indicate that these three classes cannot be easily separated by our features, so the models assign them to the most general class.
A similar explanation also applies to other low prevalence classes that fall into the broader category of Loc. B− workers. On the other hand, classes representing servers were well classified. 4.2.4. Summary The presented experimental analysis confirms that diverse device types can be classified using the representation introduced in Section 3. Furthermore, the results demonstrated that all models’ training procedures were stable throughout the week. The results also indicate that by leveraging the positional features, the GNN was able to learn higher prevalence classes and makes reasonable predictions for the classes with only a few devices.
The second experiment tests how long the models can maintain their predictive performance. All models are trained on a single day (Wednesday) and evaluated on the next six days (Thursday - Tuesday), as illustrated in Figure 7. Ten-fold cross-validation was used on the Wednesday data for training the models. Then they were evaluated on the test data from Wednesday and the whole dataset from each of the following six days. First, we examine the occurrences of devices from each class over the week. Figure 8 represents Company B and exhibits an expected pattern; the general class Protect shrinks the most during the weekend as it contains workstations of regular employees. The other classes, representing servers and IT workstations, remain stable even throughout the weekend. This pattern holds for the other two companies as well.
While expected, this observation still complicates the selection of central nodes for features representing the global structure of the graph. As explained in Section 3.2, the positional features represent the distance to a set of 50 central nodes present in the graph. The central nodes set from the Wednesday data are used as anchor nodes for evaluation on the remaining six days even though the network graph structure changes.
Figure 9 shows that for each company, the number of active central nodes substantially drops during the weekend. Since they are not present in the data, the distance to them cannot be computed and is set to − 1 (unreachable). We assume that it is one of the reasons for the poor performance of the graph-based models on weekend data. 4.3.2. Company A Figure 10 shows the accuracies for all models in Company A. The performance of the GNN model, which was by a large margin best-performing model for this company in the previous experiment, deteriorates rapidly and over the weekend falls below the performance of the baseline models. The network graph structure changes particularly during the weekend as many devices (including central nodes are inactive). Therefore, the GNN model’s performance falls very fast and only surpasses the other models on Tuesday, when most devices are active again.
The fact that all timestamps of communication logs are stored in UTC, and the private network of Company A lies in multiple diferent timezones explains why Monday still partially exhibits weekend behaviour. 4.3.3. Company B and Company C Again, due to the severe class imbalance, we report the macro averages of precision and recall, rather than accuracy, for Companies B and C in Figure 11 and Figure 12, respectively.
Since there is no train/test split for datasets from Thursday to Tuesday and the whole day’s worth of data from all active devices used instead, the low prevalence classes have more devices, and the issue with unstable confidence intervals from the previous experiment does not occur. For Company B, the models generally have a similar drop in performance over the weekend but retain their performance on other days.
For Company C, the performance of the GNN degrades notably faster than for the baseline models that do not utilise the graph structure because Company C has devices grouped partially by location, similar to Company A. This claim is further supported by the graph-based models not having this issue for Company B, with devices grouped by function and not location. 4.3.4. Summary The results indicate that the period for which the models retain their performance depends on a particular network, the nature of labels, and the model itself. For Company A, the by-far best GNN model leverages the graph structure for predictions, but it can’t handle the change of of computer networks. We encoded the global position of the graph structure during the weekend. For Company the devices in the network by calculating distances to a B, the models’ performance drops during the weekend, set of central nodes selected by their PageRank centrality but the models retain their performance the rest of the and used graph convolutional neural networks, specifidays. For Company C, the GNN model degrades, while cally GATv2 architecture. Compared with the three basethe other models retain their performance for a longer pe- line models, the GNN performed better on geographically riod. From the previous experiment’s perspective, daily structured private networks but struggled with maintainretraining of the graph models would prevent perfor- ing its performance over the weekend. To overcome this mance loss. Appendix B provides a study of the training problem, we plan to experiment with methods for classitimes of each method to support the viability of daily ifcation on dynamic graphs in the future. retaining. Improving graph-based models’ time stability will be part of our following research.
We have presented a semi-supervised approach to classifying network devices using the inherent graph structure
This appendix covers all the device features used in our research. We inspected the trafic on 14 days of 8 private networks of various sizes to design the following features that can be divided into four categories: 1. Port features 2. Time features 3. Networking features 4. Hash features The utilized device features (except the graph positional features) are standard features for capturing information in similarly structured data. Before training and evaluating the models, min-max scaling was applied to the described features.
Source and destination ports of network communication
are well-known sources of behavioural information about
the device [
Therefore, the incoming versus outgoing telemetry ratio to each service listed in Table 4 is calculated and used as features, together with the total number of unique source and destination ports observed.
The device’s activity is described by the times it communicated or was silent. Firstly, we added 24 features, representing 24 hours, and calculated percentages of the service
FTP SSH Telnet SMTP DNS DHCP HTTP NetBios
BGP LDAP
HTTPS LDAP secure FTP secure
SMB Kerberos SNMP NTP
IPP Certificate Management Protocol
ISAKMP Sun RPC
RLZ DBase webservice unassigned
SNPP Multicast DNS
SSDP Remaining well-known
Remaining registered
Remaining private/ephemeral
total communication for each respective hour. Then, the
amount of daily active hours to see how much it
communicates daily. Lastly, we compute skewness and kurtosis
of time diferences between each communication to
represent whether the device is active periodically or in bursts.
We chose skewness and kurtosis based on their success
on the time series presented in [
The networking features cover communication statistics in diferent directions and over diferent transport protocols. The following list shows the features we use: • number of all logs • number of inbound logs • number of outbound logs • percentage of outbound logs • percentage of TCP logs • percentage of UDP logs • percentage of private logs • number of IP addresses in all logs • number of IP addresses in incoming logs • number of IP addresses in outgoing logs
The hash features are only available by collecting connection logs directly on the endpoint. The binaries that initiated the connections are grouped according to their prevalence to three intervals: the top 10%, in the range 10% - 50%, and the rest. Then, the ratios of binaries the device uses over the sum of all logs in a given day are calculated. This help to identify device specific vs company-specific applications.
This section analyses the time needed for training each model. Because GNN requires additional graph-based features, we included preprocessing times in the evaluation.
Figure 13, and Figure 14 display the standalone training times for all models on Company A and C. The measured times for Company B are almost identical to Company A. The times for GNN are reported for training on the CPU and GPU for AI inference, the NVIDIA T4 Tensor Core GPU. The GPU training is around five times faster compared to the training on a CPU. model. The device-matching algorithm is a constant specific to the endpoint data. Overall, device matching and computation of device features consume the majority of the time.
For the largest network, Company C, the GNN takes
longer due to issues discussed, for example, in [
There is a considerable diference in resource consump- and faster forward and backward passes during training. tion among the models, especially on the largest network One of the key takeaways from the presented comparof Company C. For better overall assessment, all data pro- isons is that most of the time is consumed by preprocesscessing must be considered. Therefore, Figures 15, and ing raw logs rather than by model training. Furthermore, Figure 16, respectively, show the time taken to process the models do not have to be retrained from scratch but raw logs, create the respective dataset, and train each only fine-tuned for the current data, further reducing the required training time.
Adam optimizer was used for training the neural networks with the respective learning rate and hyperparameters.