In this paper, we perform a thorough match prediction analysis of our newly mined NCAA (National Collegiate Athletic Association) volleyball data set. We also investigate the comparative power of two distinct yet comparable models, namely team aggregates and player aggregates, to predict the outcome of an NCAA volleyball match. The dependent variables for both models are mainly hitting rates of serves, recepts, attacks, and assists. The output variable is the winning team. Apart from the features specific to volleyball, we also incorporated a few general match statistics. Among the multitude of Machine Learning models available for classification, the study finalizes on three primary ones viz Logistic Regression, Decision trees, and Neural Networks. Results show that Decision trees and Neural Networks perform considerably well in both the team and player models on the ROC metric and accuracy, with Neural Networks giving marginally better results. Logistic regression on team aggregates performs only slightly better than randomized outcomes, whereas, for the player model, it performs way better. In terms of model structure, player aggregates give much better classification than team aggregates with a maximum ROC of 0.98. This shows that volleyball, despite being a team sport, is intrinsically more impacted by players who make the team than the team as a whole. Our model accuracy suggests that this model can be successfully used to predict the outcome of a NCAA volleyball match.
One of the most common task in Supervised Machine Learning is the Classification task [
Our paper focuses on match prediction as a classification problem of win and loss. While most existing literature focus on mainstream sports, we sought to draw our attention to another popular sport Volleyball and fill the void and gap in the current literature. Additionally we also compare how using holistic features of team stack up against amalgamated individual player features.
In recent times, Volleyball has gained immense popularity in the world of both professional sports and recreational leagues. The sport is played at the Olympics and also has many European and American leagues associated with it.Volleyball is played both on turf and the beach. It is important to note that these are entirely diferent sports and our focus in this paper is turf volleyball of the popular NCAA (National Collegiate Athletic Association) league.
Before we dive deep into Machine Learning Research aspects, lets try to learn more about
volleyball to get an intuition of the game. We briefly explain the structure and terminologies of
Volleyball. The volleyball rules as stated in the NCAA Women’s Volleyball Handbook are as
follows:
A typical volleyball game consists of 6 members on each side of the court. The team however
consists of 10-12 players with rotating substitutions. Every time a particular side serves the ball,
there is a rotation among the 6 players so that no player serves twice continuously. The sport
can be played indoors as well as outdoors. [
Most of the basic rules remain although the size of the court and position of boundaries changes slightly. The rules for volleyball are the same for women and men with the only exception that the oficial height of the net is shorter for women. The two ends of the net must be at the same height and it cant exceed the oficial height by more than 2cm. Some basic rules of volleyball include: • A team cannot touch the ball more than 3 times before it crosses the net. • A particular player cannot touch the ball more than once.
• The ball may not be lifted, held or carried.
Two forms of scoring are observed in volleyball. A serving team loses serve if it makes a mistake, a point is only given to the serve team if the non serving team makes an error, in the other case it leads to a service change. In the formally adopted scoring mechanism, each serve results in a point to either team, its sort of a rallying mechanism. Matches are played up to 25 points and 3 games. In order to win a game a team must have a 2 point score lead. Else the score keeps accumulating even beyond 25 until either team wins. The match ends when a team wins the majority of the games i.e. 2. For this research we have used historical data for features of both teams and players to predict the winning team using the features mentioned in the following sections. Important terms and definitions used in this text can be found in the appendix section.
In [
In [
[
[
− =
[
• Attacks
• Assists
• Serves
• Blocks
The data was obtained from National Collegiate Athletic Association’s (NCAA) oficial website
[
First we made a web crawler to crawl the NCAA website to extract all the match links of the statistics for individual matches. Then the data was scraped from these links of the NCAA website using BeautifulSoup, a package available in Python. The code for this scraper is available in this repository. We use a HTML parser to parse the stats page and all the diferent tables were extracted to a Pandas dataframe and stored into a Dictionary object. The data on the website was organized match-wise. So, first the links of all matches were extracted and then they were iterated over to obtain the data for each match. We made a dictionary to hold the matches statistics of all the matches until the current match. The dictionary was used as our database for generating prior and features for the current match prediction. The key of the dictionary was either a team name to hold team stats or a tuple of (team name, player ID , player name ) to hold the player stats.
For the given match we would look into our Dictionary database and engineer the features for the current match. This ensures that there is no data leakage whatsoever into our Model predictions. After feature engineering done, we feed the current match’s data to the dictionary so that its stats can also be used as prior for next upcoming match. We discuss about the feature engineering in the next section.
It is the generation of features from the primary dataset to make it ready to be consumed by the machine learning model. It is natural to assume that to predict ℎ (Match number i of the tournament), the data that we have available to us is only till ℎ− 1.
Phase 1: In phase 1, only the overall team statistics from the past matches are used to predict the outcome of the match. Therefore, a weighted average of the diferent statistics available was taken. A weight of one suggests a similar weight to recent matches than older matches. The decay factor was taken to be 0.9 so as to incorporate a decaying efect with age/duration of the match stat. This enables giving the performance in recent matches more weight than the performances in matches played quite some time back. This can be understood simply with the help of the following expression: = = =− 1 + 0.9 * =− 2 + 0.92 * =− 3...0.9 * =0 1 + 0.9 + 0.92 + ... + 0.9 (2) Here, ’F’ refers to a particular feature corresponding to that match and ’m’ refers to the match number. Therefore, for a match number i played by a team, let’s say Arizona, the features for that match would be a weighted average from the Arizona’s previous match to Arizona’s first match in that season.
Phase 2: In phase 2, player-wise statistics were used for the study. For each player of a particular team, the features of that player were engineered from their performance in the previous matches playing for the same team. The features were weighted in the same way as in phase 1.
In order to predict the wins in a volleyball match, we decided upon two diferent paradigms:
First, A structure that lays emphasis only on the team features and decides the match outcomes
on the team statistics without giving much of a direct focus on the players. The other approach
is to look at the players being the strength of the team and incorporate more directly the impact
of the stronger players, rather than averages. The final features of both the approaches are
calculated as historical averages. There are broadly six skills in volleyball that are considered
crucial to a player/team’s strength. Four of these are considered in this paper due to their relative
importance: Attack, Assist, Serve, and Recept [
Start Crawl across NCAA website and scrape all match links for
years 2011-15 Match Links Dataset
from 2011-15 Iterate across and links and scrape the tables with Beautiful soup's HTML parser
Current Match
Features
Generated? In our dataset 0 implies team1 won and 1 implies team2 won. We can make total 1s equal to total 0s by switching the teams whenever one of the class is imbalanced. This is done by us using a random number algorithm
Update the
Database after features generated for current match yes
Historical dataset stores all
match stats about a particular player and team.
We use this historical information to make features for current match
Historical Database Decay data proportional to how old the match is and take weighted average
Do Feature Engineering for Team
Data
Decay data proportional to how old the match is and take weighted average
Do Feature Engineering for
Player Data Team Feature Data in csv format
Player Feature Data in
csv format Balance the classes in dataset
Balance the classes
in dataset Balanced Team features data
Balanced Player features data later. The first model that we choose is Logistic Regression. Our label is a binary variable and hence, the Logistic Regression model calculates the probability in the following way: = −
1 1 + − ( 0+ 1* 1+...+ * )
Here, is the parameter set for the model. The probability of Y belonging to label 0 can be just calculated by ( = 0|, ) = 1 − ( = 1|, ).
As our second machine learning model, we choose Artificial Neural Networks also called
feed-forward neural networks to be more precise. [
The third model is Decision Trees. A minimal cost complexity approach is used for pruning in the model. The splitting criterion used is - randomly initiate the threshold for all features and then iterative to find the best split. This helps reduce over fitting.
Overcoming class imbalance Due to some cultural or circumstantial reasons, the NCAA data inherently had stats where the second team on the list won 90% of the matches. Upon further analysis we found no reasonable argument for the same. Also, most of the NCAA matches were played in neutral venues. To remove this bias from our data set we balance the number of 0’s and 1’s by randomly shufling the ordering of the teams so that there is an equal probability of either team winning. The shufling has lead to no loss of generality and the data set ends up being balanced. 4.1. Phase 1 In phase 1, we focus on team aggregates as a whole to predict wins. Individual match data was collected from the NCAA website. The train set consisted of six features which are calculated using the match statistics.
The data is cumulative in nature, in the sense that every following year contains the information cumulatively up to, but not including that year.
The following features are used:
• Attack PCT : Attack pct measures the average attacking power of the team. Attacks
are the most important way for an ofensive team to win points. Attack pct is directly
proportional to the win rate [
• Serve PCT : Serve PCT is a measure of the number of serve aces by a team.
= (5) • Assist PCT : A player is awarded an assist if he/she passes the ball to a teammate who then closes in on a kill or attack. (3) (4) (6)
• Recept PCT :How well a team handles a potential serve ace is measured by the recept PCT.
= • Set Win Ratio : It is the ratio of total set wins by all sets played.
=
(7) (8)
One important thing to note is that the features mentioned above show a high level of
correlation with each other as shown in Figure 2 . This can be attributed to the fact that a
proficient player might possess more than one skill to a reasonable extent [
Similar to [
= + + − − + + + + where BD is Blocking digs, BS is Block solos, BA is Block Assists, BE is Block Errors, BHE is Ball handling errors. Xij refers to the player j of a particular team i.
Ace is a serve which lands in the opponent’s court without being touched, or is touched, but unable to be kept in play by one or more receiving team players, resulting in a point for the team serving. Since Ace is a special kind of serve, this ratio is a representative of the player’s serving skills and hence used as a feature.
ℎ =
(9) (10)
The authors collected data of 299 matches spread across five years. For a particular year, there
are fifty to sixty matches. But, using player-wise data leads to an explosion in the number of
features because there are five features each for the twenty players who are going to play in
that match (Ten per team) summing up to a total of 100 features. This is a severe problem [
It can be observed that all the five features for the players are eficiency ratios. Therefore, they are already normalized between zero to one. To reduce the number of features for prediction and to not lose important information, a single metric is allotted to every player. The metric (player-Score), acts as a proxy of the player strength and provides information on how valuable is the player to the team. It is calculated in the following way: = + + +ℎ+ (11) 5
Here, similar to conventions followed above, j represents a particular player of a team i. Hence, it is just a mean of all those five features. These scores are generated for every player who is (predicted to) play in that match. After this is done, the data set is transformed to a new lower-dimensional data set. The new data set has the five featured mentioned above (of which the mean is taken) for three players of each team. The three players are said to be the representatives of the team for that match.
The first representative player’s stats are an average of the best three players of that team on the basis of the calculated playerScore. Similarly, the second representative is an average of the three players who have medium strength - The fourth, fifth and the sixth best players for that team. And the third representative is an average of the poor performing teammates, having the least four playerScores. These three representatives are calculated in the same manner for the second team. Now, for each match we have a total of 30 features containing the features of 3 representatives for each team.
This would allow us to capture important relationships between the match winner and the 3 representatives. Using the coeficients and the importance of features in case of logistic regression and decision tree respectively, we can empirically conclude relations between the match winner and the best players, the average players and the worst players.
In this section we give details about the architecture of our models. These are the three models that were constructed for training on data set for our prediction. Each model was tuned for the best hyperparamters.
1. Logistic Regression model 2. Decision Tree Classifier 3. Artificial Neural Network
Logistic Regression A tuned Logistic regression was used as a baseline for model training. A limit on the max iterations was set to 100000. The optimizer is set to limited memory BFGS(lbfgs) for our model.
Decision Tree Classifier Decision Tree Classifier from the sklearn library was used to implement the decision tree classifier. A critical factor is that with such limited data points and without any hyperparameter tuning, the tree over-fits the training set completely. We set a particular value for ccp_alpha, which is the hyperparameter for cost complexity pruning in Decision Tree provided by the sklearn library. To observe the variation in impurities in the leaves with the changing ccp_alpha, we plot the following graph.
We set ccp_alpha at 0.06 in our final model so that the model does not overfit and is robust. The model uses entropy as the criteria to judge the quality of split. The class weight is set to "balanced" so as to learn both the class labels equally.
Artificial Neural Networks
After firm experimentation on regularization in our ANN models, we zeroed in on the architectures which have been mentioned in the results. We use Adam optimizer and disable shufling to prevent data leakage as our data is time series in nature.
Since this is a classification task we have taken the ROC-AUC metric to analyze our model. Second metric we have used is F1-score, F1 Score is the harmonic mean of precision and recall. F1 score conveys the balance between the precision and the recall. We have used data spanning from 2011 to 2015 and since our data is dependent on time and the sequence in which matches were played, we have used the following train test split:
Eg: here, data from matches played in 2011, 2012 would be used to predict for matches in 2013. We take the average of ROC-AUC score obtained in each of the above 4 cases. We have taken care that within the same season also the match data should not be shufled and is ordered according to the dates when they took place.
In this section we compare the results of the models developed using Team Data and the features mentioned in section 4.1.
We develop a Logistic Regression Model for Binary Classification. We set the max iterations as 100 and set the class weight parameter as "balanced" to automatically adjust weights inversely proportional to class frequencies in the input data. The value of C has been chosen by tuning is across various values to get appropriate regularization(200 values on logarithmic scale). The output labels would be 1 or 0 depending on which team is predicted to win(0: team1 wins, 1: team2 wins)
We use the Decision Tree Classifier and set the splitting criterion as gini and set the splitter as ’random’, so that all features are sampled randomly according to the feature importance. As we have limited data, we want to ensure that the decision tree does not overfit. To ensure this we have tuned the max_depth and min_samples_leaf as 10 and 20 respectively. The ccp_alpha parameter here has been chosen as 0.01.
We use Sequential Model provided by Keras Library for the Artificial Neural Network. The 5x2 features from both teams i.e. 10 units form the input layer. We use the ReLU (Rectified Linear Unit) activation function at both stages to learn a non-linear mapping for classification task. The final output is passed through a Sigmoid function- which will give an output in the range (0,1) denoting the probability of the team winning for our binary classification task. We train the Neural Network for 100 epochs. We set shufle = FALSE as we want prevent data leakage as our data is time series in nature.
Input Dense Dense Output 10 units → 15 units,Actn:ReLU → 25 units,Actn:ReLU → 1 unit,Actn:Sigmoid
In the above 3 models, there is a similar trend in the ROC/F1-score vs the Split being trained on. As the training data increases, the metrics of the model improve as the model has learned on more data.
In this section, we document the results obtained from the diferent models trained on playerwise statistics for a given Volleyball match.
Similar to Phase 1 Logistic regression on Team data, we tuned and trained a Logistic Regression Model for Players data to set a baseline . We set the class weight parameter as ”balanced” to automatically adjust weights inversely proportional to class frequencies in the input data. We tuned across multiple C values (2000 of them divided on a logarithmic scale) to get the best regularization .
The Decision Tree is trained with ccp_alpha at 0.06. The class weights are set to balanced so that both the classes are learned equally. The criterion to decide the quality of a split is taken to be entropy.
We used player vectors of 30 features as input to the Neural Network and set shufle = FALSE as we want prevent data leakage as our data is time series in nature. We trained for 10 epochs with Adam optimizer. Below is the architecture which gave the best results after tuning.
Input 20 units →
Dense 20 units, Actn:ReLU , Dropout:0.25 →
Dense 25 units, Actn:ReLU, Dropout:0.25 →
Dense 30 units, Actn:ReLU , Dropout:0.25 →
Output 1 unit, Actn:Sigmoid
Split1 Split2 Split3 Split4 Mean
In the above three models there is a clear trend of better metrics with more and more data with every passing year.
The results that volleyball despite being a team sport , results are intrinsically more impacted by players who make the team than the team as a whole. In other words , models trained on Player data contain finer statistics than model trained on team data. This shows that in sports prediction bifurcated chunks of features which make the entity which is team gives better information to the model and is a better predictor.
Our classification results achieved better performance than strict guessing in all cases, with prediction ROC’s ranging from 0.74 to almost 0.98 in some cases. In phase 1, the roc scores are 0.64, 0.75, 0.84 for LR, DT and NN’s. Similarly for Phase 2 the corresponding scores are 0.98, 0.95, 0.98. Neural Networks perform better in both phases. The diferences in accuracy are primarily due to the diferent approaches that we have chosen for this classification task. The aggregate player approach seems to have picked up the key causes that result in a team win as it incorporates significantly more information in its features. Although our model parameters were finalized through a series of experiments, we are aware that more specialized models could result in higher accuracy. Some general features such as average age of players, average age of the team, number of new players etc can be used as well. We were unable to use the same due to a lack of relevant data.
The metric for accuracy used was ROC AUC, as it is not biased towards the size of test or evaluation data. While accuracy is measured on predicted classes, roc auc is measured on predicted scores which makes roc scores and f1 scores better for classification tasks. A high accuracy moreover could be due to over-fitting.
In phase 1, the mean ROC is highest for Artificial Neural Networks as they have the ability to learn and model non-linear and complex relationships between inputs and outputs. We observe Decision Tree performs much better in split4 when it is supplied the maximum test data.
Further extension of this model could use the extensive NCAA data available to make the current model more robust and versatile. Moreover, it could also incorporate a home and away team feature. In the current literature, we were unable to do so because the NCAA games are not necessarily conducted in the home or away grounds. We believe this is a crucial factor that must be used for sports predictions. We can also experiment with Recurrent Neural Network (RNN) architectures to learn from the temporal property of the data.
An essential point of consideration for any probabilistic model is the inclusion of prior
probabilities for all possible outcomes. One way to measure priors as mentioned in [
The model that we have trained do not capture the synergistic relations between the diferent
players. Although, Artificial Neural Networks might capture these relations implicitly, no
inferences can be made from the trained neural network on these synergistic relations between
the players. [
These network-based approaches move away from the conventional machine learning methods, and include richer information into the model. Useful features can be generated after network analysis which can be helped generate robust models for the same conventional machine learning algorithms.
While we used an algorithm of sorting and merging players in sections of 3, 3, 4 K means could
have been employed to cluster similar players together and replacing the vectors of clustered
Player with one single vector. [
We have chosen a decay factor of 0.9 with respect to the previous matches. But, technically the selection of the decay factor is in itself a search problem. Further research can be done to understand how the decay factor should vary within earlier matches of same season and matches from previous season. Here we have considered a geometrical decay factor. There are other questions like should this decay factors be varying for the past years? For example, should matches from 2 years back have the same decay factor as the matches from the past year. Most teams in NCAA play only for about 1-2 matches in a season. Hence, impacts of past 2-3 years can be seen in the data if we just use a simple decay factor without analyzing these things. The decay factors can also be adjusted on the basis of the volatility of players’ skill. There are a lot of possible future directions this paper can be extended to and it will be interesting to see the impacts of them on the predictability.
In conclusion, we believe taking into consideration our limiting factors such as limited data availability, class imbalance, and inability to use cross-validation or shufling as we were constrained by time-series nature of the data, the aggregative player model implements a sound classification task of predicting a volleyball win and can be used successfully for the given task.