This paper provides an overview of the research activities in the area of Artificial Intelligence applied to Network Function Virtualization (NFV), carried out by Vodafone, jointly with Scuola Superiore Sant'Anna of Pisa. Artificial Intelligence techniques have been used on system-level data gathered from Virtual Machines (VMs) composing a multitude of Virtualized Network Functions (VNFs), to tackle a number of problems: from trafic forecasting for capacity planning and optimization, to the of-line analysis of the daily behavior of metrics to identify possible anomalous patterns, to a Near Real time (NRT) approach for metric prediction and anomaly detection, so to trigger prompt reaction of operators of the infrastructure and services. These problems become particularly challenging in the context of the Vodafone infrastructure, spanning across several data centers for NFV throughout a dozen European Countries.
1. Introduction
munications, where network operators are increasingly
adopting virtualization technologies and principles from
the area of Cloud Computing, as key ingredients to create
lfexible and scalable network infrastructures. This led to
the so-called Network Function Virtualization (NFV) [
(SOM) to analyze patterns of VM metrics in data centers for NFV, to provide a visual understanding of the main behavioral patterns, promptly detect anomalies. This technique can perform a joint analysis of system-level metrics obtained from the infrastructure monitoring system from individual VNFs (VNF metrics). These metrics are In our technique, a VM is observed through its moveobtained from the NFV infrastructure manager, VMWare ment among the best matching unit (BMU) during the vRealize Operations 1, and the monitoring subsystems analysis time frame. Any changes in the BMU that are disof the virtualized services. By jointly analyzing these tant from the previous location could indicate anomalous metrics, we can gain a more comprehensive understand- behavior and trigger an alarm. This enables an operator ing of the major behavioral patterns of VMs and detect to focus on a specific set of VMs and their hosts, and consuspicious (anomalous) behaviors. duct a further analysis that would be too time-consuming
SOM-based clustering was performed jointly on a set or impractical for the entire infrastructure. of input metrics, analyzing monthly data at a 5-minutes Additionally, we provide a mechanism for automated granularity (288 samples per day per metric per moni- detection of potential suspect behaviors. A simple tored VM), resulting in several GBs of data per month for threshold-based alert is triggered whenever an input sama specific region. ple is associated with a neuron that has a quantization
Figure 1 shows the workflow used to transform the error greater than the specified threshold (i.e., it is too input INFRA metrics. First, the raw data undergo pre- far from its BMU). These samples are likely to represent processing to address any data-quality issues and to re- uncommon behaviors and are marked as misclassified. tain only relevant metric information. Next, the input The misclassified samples can be regarded as suspect samples for each VM are constructed by consolidating or anomalous patterns that require further inspection. the individual metric contributions into a single vector The misclassification mechanism can also immediately for each pre-defined period. The considered steps are: notify operators of potential misconfigurations where a i) normalization: this was done scaling each daily time- too small SOM grid size has been chosen, leading to an series pattern by either subtracting its mean and dividing excessive number of misclassified time series. by its standard deviation, or normalizing to a range of values between 0 and 1 using the historical minima and 2.1. Grouping neurons maxima values observed for each metric; ii) missing value treatment: to address missing data and significant difer- A noteworthy observation from using the SOM-based ences in metric magnitude, a data imputation strategy classification is that when employing relatively large consisting of simple linear interpolation is performed; iii) SOM networks, the training phase often resulted in mulifltering: the input data are filtered on the k specified met- tiple adjacent SOM neurons capturing very similar behavrics and partitioned to have a sample for each metric, VM, iors. This aligns with the topology-preservation property and period. Each input vector to the SOM is a concatena- of SOMs, which maps similar input vectors in the input tion of k vectors related to the pre-processed time-series space to adjacent neurons in the SOM grid. Although of the k metrics for 1 day for one of the considered VMs. this can be controlled to some extent using various neigh
Once the training phase is completed, the SOM is used borhood radiuses, data center operators need to view a to identify the best matching unit (BMU) for each input group of adjacent neurons with similar weight vectors as sample, providing the clustering functionality. The BMU a single behavioral cluster. To address this, we incorpois the neuron that has the least quantization error when rated a straightforward clustering strategy after the SOM compared with the input sample. This output can be processing stage, which combines neurons having weight used by a data center operators to visually examine the vectors at a distance lower than a specified threshold into behaviors captured by the trained SOM neurons, and the same group. Consequently, our technique enables the identify suspect or anomalous VM behaviors. consolidation of similar clusters based on the distances between the representative vectors of SOM neurons, reducing the likelihood of triggering an unnecessary alarm
(e.g., frequent movements of a VM over time between two similar neurons) and facilitating the interpretation of results by human operators.
As an example, we have selected a data set containing
CPU and NET metrics for several VMs in April 2020.
Fig. 2 shows how the 5 × 5 SOM has classified the whole
behaviors of diferent VMs in 5 groups i.e. red, orange
which are regarded as high-working level groups, green
and brown which are low-working and gray which is
almost a flat neuron. Also, the behavior of each VM and
its possible change during the whole month is shown in
Fig. 3. This way, we designed an alerting system for VMs
based on their daily behavior. Namely, by considering a
period of time, an alert is raised once the daily behavior
of a VM has been classified once in a diferent group.
The results of such an alerting system, called ”Strong
Alerting System” (SAS), is shown in Fig. 4, where the
dark green cells are the alerts. However, SAS showed
to be prone to create a lot of false positives, since even
one behavioral change raises the alert, but, as visible
in the figure, several changes happen recurrently over
week-ends, so they are to be regarded as non-anomalous
changes. To overcome this issue, we also defined a ”Weak
Alerting System” (WAS), where an alert is raised when
a group change occurs that is not among the weekly
changes occurring every week-end. Interested readers
can find more details in [
detection techniques are utilized to identify issues within the infrastructure by examining the vast amount of data available through the monitoring subsystem. Real-time anomaly detection, or NRT, aims to accomplish this task promptly as soon as new data is obtained at run-time. We are dealing with metrics to be analyzed in real-time from all the NFV data centers located in 11 EU countries. The main objective is to identify anomalous points in the resource consumption and application level metrics of VMs/VNFs, which are monitored in the NFV and cloud management frameworks. Therefore, we require a scalable design, which is explained below.
(GCP) environment, using the Cloud Big Table3 service as a reliable NoSQL storage for the gathered time-series. Meta-data related to all active VMs in the Vodafone virtual network infrastructure is stored in a separate SQL database, including their unique identifiers and timestamps of creation, termination or other relevant events.
The raw metrics data collected every 5 minutes for each VM are merged into a single vector for a given period and cleaned before processing with diferent algorithms for anomaly detection. The system is modular, allowing the configuration of various ML/AI techniques that comply with a simple interface, deployed as Google Cloud Functions4. These functions are triggered periodically using the Google Tasks service5, with the output being an anomaly score for each new data point injected into the data processing pipeline since the last activation.
In the post-processing phase, single anomalous points followed by non-anomalous points are ignored, while sequences of three or more timestamps marked as anomalous are saved in a persistent storage database accessible to operators through the Grafana framework6.
In order to perform NRT anomaly detection (AD), we
use two main techniques: Prediction-based AD, where
values output by a prediction model are compared with
the actual samples, and, if a given threshold is exceeded,
the samples are considered anomalous; ML Algorithms
designed to directly identify anomalies/outliers, such
as Isolation Forest [
We used two main methods to perform NRT anomaly detection based on predictive models: Long-ShortTerm-Memory (LSTM) auto-encoders, and Simple Median (SM), a much simpler model based on statistical and mathematical relations among the values of the data-set, i.e., based on the ”averaged” behavior of the previous days of each VM and calculated based on a statistical median.
whose behavior difer from the behavior exhibited previously by the same VM, as visible in Fig. 6.
We have performed comparisons among the accuracy in AD obtained with diferent methods and algorithms. Some AD algorithms, like Isolation Forests, were able to spot U-shape anomalous intervals like those shown in
Fig. 6, only when occurring at the borders of the statistical distribution of the samples, failing to detect them in several cases, in our data sets. However, a vectorial extension of Isolation Forests was able to spot at least the beginning and ending intervals of these anomalies.
For the predictive models, we identified a number of
scenarios in our reference data set as particularly critical,
because they were including multiple diferent
anomalies occurring in the same day, and/or in consecutive
days, sometimes spanning 3-4 consecutive days, making
the analysis more challenging. In this case, the Simple
Median detector we realized outperformed LSTM
autoencoders both in spotting diferent anomalies and
producing fewer false alarms. Fig. 6 shows specifically that
SM has been able to spot all anomalous points of two
diferent anomalous intervals for a particular VM on 30th
of January without any false positive detection. Details
are skipped for the sake of brevity, but the interested
reader can find additional details in [
At Vodafone, the deployment of an NFV Infrastructure consists in allocating computation workload, in the form of Virtual Machines (VMs), taking care of not exceeding the available hardware resources of the servers, considering the logistic limitations, and dealing with afinity/antiafinity constraints on the workload. The optimal resource allocation problem has been tackled using both classical optimization, and a Genetic Algorithm.
are employed when optimality guarantees are needed.
Optimal placement problems are usually encoded as
Mixed Integer Linear Programs (MILP) or, as Boolean
Linear Programs (BLP), as done in [
mon approach to the resource provisioning problem.
These are usually ad hoc algorithms designed to
provide a solution, following simple rules that depend on
the specific problem to be solved. A lot of efort has been
placed into developing Heuristics for resource allocation
problems [
ware Defined components on their network, a hybrid
approach that exploits Computational Intelligence has
been pursued. The success of AI techniques for solv- Figure 7: Generic Structure of a Genetic Algorithm.
ing similar placement problems is well reported in many
other works, such as [
by Vodafone, has been set up to evaluate the proposed approach. The goal was to allocate the Virtual Machines to Genetic Algorithms Genetic Algorithms are based on the minimum number of Hosts. The results are compared simple heuristics and can take advantage of the knowl- with the ones obtained from a classical BLP approach and edge of the problem, having the possibility to avoid local a simple First-Fit heuristic. As expected, the proposed minima, and thus are more likely to reach good-enough approach achieved a trade-of between the quality of the solutions. By tuning the algorithm hyper-parameters, it solution, in terms of used Hosts, and the time to get the is possible to achieve a good performance, in terms of so- solution. In Fig. 8 the solutions of some representative lution optimality, and still get a solution time comparable instances, for a diferent amount of VMs to be placed, with the simple heuristics. are reported. For large problems with thousands of VMs
Instead of operating on a single solution, Genetic Al- the Heuristic approach delivers a sub-optimal solution, gorithms evaluate a population of diferent solutions, whilst the Genetic Algorithm’s one is comparable with iteratively evaluating every single solution and propagat- the optimal one returned from the MILP problem. As ing a subset of the population (active population) selected far as the computation time is concerned, Fig. 9 reports with some criteria. A schematic representation of the the time necessary to obtain the solutions reported in algorithm is shown in Fig. 7. Often, the criteria to sort Fig. 8. In this case, the pattern shows that the Heuristic and select candidates are just how good the solution is, is the fastest approach, while the MILP approach takes but there are cases where the selection also considers the longest amount of time to return the solutions. The the population variability and some specific features. Be- Genetic Algorithm stays in between. For large problems, tween iterations, the selected solutions are also blended the solution time of the Genetic Algorithm is still considtogether or altered to generate new individuals that can ered acceptable by Vodafone operators, whilst the MILP hopefully have the good properties of the parents even- is considered too slow. tually improving with respect to them.
In the specific case of the approach used at Vodafone, 5. Conclusions the Genetic Algorithm is based on a First Fit heuristic where the processing order of the VMs to be placed is This paper provided an overview of how AI and ML techthe optimization variable of the Genetic algorithm. That niques are being used within the Vodafone NFV infrasis, the algorithm will search for the processing order tructure to ease and enhance a number of data-center that will give the best result once placed by a First-Fit. operations related to workload prediction, anomaly deThe generation of new candidates is implemented by tection and capacity planning. mutating existing solutions. The mutation randomly