Network managers that succeed in improving the accuracy of client video service level predictions, where the video is deployed in a cloud infrastructure, will have the ability to deliver responsive, SLA-compliant service to their customers. Meeting up-time guarantees, achieving rapid rst-call resolution, and minimizing time-to-recovery after video service outages will maintain customer loyalty. To date, regression-based models have been applied to generate these predictions for client machines using the kernel metrics of a server cluster. The e ect of time-varying loads on cloud-hosted video servers, which arise due to dynamic user requests have not been leveraged to improve prediction using regularized learning algorithms such as the LASSO and Elastic Net and also Random Forest. We evaluate the performance of load-adjusted learning strategies using a number of learning algorithms and demonstrate that improved predictions are achieved irrespective of the learning approach. A secondary bene t of the load-adjusted learning approach is that it reduces the computational cost as long as the load is not constant. Finally, we demonstrate that Random Forest signi cantly improve the prediction performance produced by the best performing linear regression variant, the Elastic Net.
Streaming video content over wired and wireless communication networks will
be a major contributor to future internet tra c as can be inferred from [
Fig. 1 depicts the set-up of the system under study in this paper which is representative of real world systems. It is composed of a cloud-based server infrastructure servicing dynamically and time-varying client requests received over a network. Server resources are shared between multiple clients with the video service (RHS) delivering video to target client machines (LHS). The number of users accessing the server resources changes rapidly and this poses a challenge in predicting the target client's video quality. A video server of this form must be able to handle time-varying loads as users can start and stop videos at arbitrary times simultaneously.
We seek a model that characterizes the e ect of the time-varying loads on
the client's video quality provided we have knowledge of the kernel metrics of
the server delivering the video. Both the client machine and the server clocks
are synchronized to match up observations. Samples are drawn from the client
and server machines every second. A VLC media player services the
Video-onDemand (VoD) requests in [
We contribute adaptive learning techniques which reduce the computational
complexity, and provide more accurate predictions over the baseline approach in
[
This paper is organized as follows. In Section 2, we introduce the load-adjusted learning technique and the Machine Learning (ML) techniques. Section 3 introduces the model tting procedures and the evaluation framework. In Section 4 we evaluate the e cacy of each of the approaches. Section 5 places our contribution in the context of related literature and we conclude our work in Section 6. 2
The server in Fig. 1, collects device statistics, x using SAR. The number of active clients, the load signal at time i can be measured with the TCPSCK feature of x. A load generator dynamically allocates client requests for video to the server under two load patterns, a periodic-load pattern and ashcrowd-load pattern. In the periodic-load pattern, clients are started following a Poisson process with an average arrival rate of 30 clients per minute. This arrival rate is modulated using a sinusoidal function with a period of an hour and an amplitude of 20 clients. The ashcrowd-load pattern starts clients with a Poisson process with 5 clients per minute average arrival rate and peaks at randomly created events at a rate of 10 events per hour. During ash events, the average arrival rate swaps to 50 clients per minute for a minute and then gradually reduces to 5 clients per minute within the next 4 minutes.
Using the device statistics computed from SAR, service-level metrics can be computed at the clients. In this paper, we are interested in predicting the (i) Video Frame Rate (VFR), the number of displayed video frames for any time i; (ii) Audio Bu er Rate (ABR), audio bu ers played for time i; and (iii) RTP packet count, the number of received RTP packets in time i. Fig. 2 illustrates a plot of the RTP packet count and the TCPSCK count, the load proxy recorded over a period of 15000 seconds for both load patterns. As can be inferred from Fig. 2, the TCPSCK count rises with an increase in load and may result to a reduction in the RTP packet received at the client because the system has limited resources. 2.1
The e ect of concurrent requests on the server kernel are examined. In Fig. 2, we plot the RTP packet count recorded over 15000 seconds and the TCPSCK kernel parameter over the same period of time. From visual inspection of the plot, the RTP packet count has the same periodic pattern as the TCPSCK. It is also evident that as the load increases, the TCPSCK count increases and may lead to a decrease in the number of RTP packet counts at the client because the system does not have unlimited resources.
Load-adjusted model: Let n represent one video resource currently being
used by a client. A server response with respect to its kernel metric, the n-th
feature, to one request for a video at time i is the sum of the resource a user has
and some deviation signal speci c to a feature [
where i 2 Z; xi[n]; n 2 R:
An additional request for resources by the current user or a new client would invoke a feature response of the form:
K(i) xi[n] = nK[i] + X i[n; k]: k=1 The load signal nK[i] denotes the number of active users at time i times the resources one user uses, n. The TCPSCK is K[i].
Un-adjusted Learning (UA): Previous attempts at predicting service-level metrics from device statistics do not model the e ect of the time-varying load. Problem Statement: Our objective is to learn a model that predicts servicelevel QoS metrics, y: RTP packet rates, video frame rate and audio bu er rate using the features xi[n] given a time varying load K[i]. Using both the ashcrowdload and periodic-load traces, we test the hypothesis that a learning algorithm which takes the load value into consideration produces better predictions than models obtained using algorithms which ignore the load. They assume that K[i] is constant C. 2.2
In this section, we brie y introduce the di erent ML techniques we adopted for our experiments.
Linear Regression: We start with Linear Regression (LR), a baseline for many ML techniques. The LR models a linear relationship between the metrics we want to predict y, the dependent variables and the independent variables of predictors x as a linear function of the form: y^i =
N
X xi[n] n
n=1
where xn[
Random Forest (RF): We apply a non-linear method, the Random Forest
algorithm, which is an ensemble method which builds multiple decision trees
and consolidates their results to obtain stable and more accurate predictions. In
simple terms, the RF estimates y^ for the metric y using the average predictions
from a large number of regression trees [
We perform model computations using the traces made publicly available in [
Un-Adjusted Method (UA): For the UA learning, we adopt the technique
used in [
The regularization techniques, RR, LASSO and EN required a method for selecting the regularization parameter, , for the penalty function. RR, LASSO and EN use an `2-norm, `1-norm and combination of both norms as a weighted (by ) penalty term. The entire path for the results for these models was calculated using path-wise cyclical coordinate descent algorithms. Computationally e cient and e ective approaches for evaluating these convex optimization problems were implemented using the glmnet package in R.
To obtain the value of , we employed 10-fold cross-validation (CV) approach for both learning approaches. This value was used in subsequent learning and prediction experiments. The 10-fold CV was implemented for training the models and during testing using both traces. Di erent values for were determined for both the UA and LA algorithms. A sequence of values between 0:0001 and 1 was selected and cross-validation applied to select optimal values for the regularized models. EN outperformed the other three linear methods evaluated and was adopted for our model performance comparison between the LA and UA models. Load-Adjusted Method (LA): We obtain subsets of the entire data set for which the load value, the TCPSCK, is xed and has more than 500 samples. To ensure that we have enough data to split between train and test samples, we use the top subsets with the most samples in them for the LA learning. Using the validation set approach, we divide the traces into two; 60% of the trace in the training set and 40% in the test set. We apply the same crossvalidation procedures used for the UA models to nd the best values for the EN algorithm using the LA method. We learn EN models for each subset of the data based on the TCPSCK value. We then apply EN to the traces to learn UA prediction models for VFR, ABR and RTP packet rate. We refer to these models as Un-adjusted Elastic Net (UA-EN) models. We learn Load-adjusted Elastic Net (LA-EN) models for our service level metrics with the same number of samples as was done for the UA-EN models.
Using the same samples used for the UA-EN and LA-EN models, we then apply Random Forest to learn Load-Adjusted Random Forest (LA-RF) and Unadjusted Random Forest (UA-RF) models for the service-level metrics.
All models are evaluated in terms of two accuracy measures. The rst is the Root Mean Squared Error (RMSE), computed as q n1 Pin=1 (yi y^i)2. The best model is the model with the lowest RMSE. The second measure is the R-squared which is essentially a statistical measure that explains the goodness of t of our regression models. R-squared achieves this by comparing our regression models with a baseline model, one which is simply the average of of observed responses of the dependent feature. The R-squared is computed as R2 = 1 SSSSET where the Sum of Squared Errors (SSE) in our model is computed as Pin=1 (yi y^i)2 and the Sum of Squared Total (SST) of our baseline model is computed as Pin=1 (yi yi)2. The model with the highest R-squared value is the best, and a model with an R-squared value of 1 is a perfect model. We report only the test RMSE and R-squared values.
We present three results: (1) We compare the performance of the four linear models we examined using the UA approach. (2) We examine the performance of the best performing linear model with the non-linear model, the Random Forest algorithm. (3) Finally, we compare the performance of our LA models with the UA approach.
UA Linear Models: Table 2 lists the performance of the linear models LR, RR, LASSO and the EN on the test data for both the Periodic-load and Flashcrowdload traces using the UA technique. 1. The EN gives the best result out of all four linear models for both traces. The RMSE for all three metrics predicted is lowest for the EN and the R-squared is highest for the EN models in both traces (boldfont in table). 2. The EN performance is closely matched by the LASSO and the LR for both traces across all three metrics. The EN o ers the best prediction accuracy due to its ability to overcome the limitations of LASSO by automatically tuning the loss function based on the data. The results indicate that the EN does well in both traces but seems to o er lower RMSE values for the VFR and ABR using the Flashcrowd trace. 3. The RR algorithm o ers the worst predictions across all three metrics for both traces.
Un-adjusted Method
Periodic-load Trace
Random Forest 4. The performance of the Random Forest algorithm using the UA approach is listed in Table 2 for both load traces. The RF algorithm o ers a big improvement in RMSE and R-squared over the EN. The RF performance gain over the indicates that non-linear methods perform signi cantly better than linear models as can be inferred from the results.
Comparison of LA and UA: We have listed results for EN and RF models learned using the LA and UA approaches in Table 3. 1. The LA models for EN and RF outperform the UA models for both the Periodic-load and Flashcrowd-load traces across all three service-level metrics. The LA-EN estimates are over 5 audio bu ers/second better than for the UA-EN in both load traces; the LA-EN estimates for VFR and RTP indicate similar improvements in both load traces. The LA-EN o ers similar performance gains for both traces. 2. The RF algorithm o ers better RMSE and R-squared values for the LA technique than the LA-EN. The LA-RF shows a big improvement in estimates. We compare the average RTP prediction to illustrate what the RMSE values imply. For instance, the LA-RF estimates for the RTP using the Flashcrowd trace indicate that the values lie between 90 to 449 RTP packets. The UARF estimates for the same trace o er estimates between 8 to 347 RTP packets/second. True RTP values lie between 83 to 545 RTP packets/second. Expressed in percentages, the average improvement in prediction performance is 50% to 60% better for the LA RF learning. 3. The LA models in the Flashcrowd-load trace o er better prediction metrics than the Periodic-load. Fig. 3 illustrates the accuracy of the LA predictions for the RTP packet/second for both traces for comaprison with the UA predictions.
Flashcrowd LA-EN
True Predicted True Predicted True
Predicted 1 Periodic LA-EN
400 1 Flashcrowd LA-RF
400 500 y 100 500 y 100 500 y 100 500 y 100 1 800 800 800 500 100 500 100 500 100 500 100
Flashcrowd UA-EN
True
Predicted 1 Periodic UA-EN
400 True Predicted True Predicted True
Predicted 1 Flashcrowd UA-RF
400 1 Periodic UA-RF 400 800 800 800 800 1 Periodic LA-RF
True Predicted 400 400 Time (s) 800 1
400
Time (s)
Yanggratoke et al.in [
The authors of [
Our results are of relevance to networking professionals. Our load adjusted
approach is computationally cheap than UA learning. We consider subset of the
data based on the load value which improves prediction accuracy and reduces
computation. From the perspective of the network service provider, the authors
of [
There is a signi cant momentum behind in the concept of Software De ned
Networks (SDN) which will lay the foundation for our future work, particularly,
how we will deploy our learning engine. The authors of [
The LA improvements in performance makes no additional assumptions about the data. We will investigate how weakened forms of the independence assumption made by these models can improve prediction.
We introduced a method for improving predictions of service-level metrics using a load adjusted learning technique. We provided evidence that the EN algorithm provides the best prediction performance among the linear regression variants using the baseline UA approach. We also presented evidence which shows that the LA learning algorithm improves the UA prediction performance for all three metrics under study. We further demonstrated that the Random Forest predictions outperform the EN estimates using the load adjusted approach. The LA method o ers signi cant improvements in the prediction accuracy and reduces the computational requirements of the system delivering the resources. Acknowledgement. This publication has emanated from research conducted with the nancial support of Science Foundation Ireland (SFI) under the Grant Number 15/SIRG/3459.