The need to organize prevalent multimedia in our day-to-day activities, presents us various aspects of video importance to be taken into consideration, like video aesthetics and interestingness. Memorability based organization of media is highly efective in cases where media is required to have a long positive retainability in one's memory, like in the field of advertising or educational content creation. Keeping our target group in mind, we have decided to pursue an ensemble-based approach for our model for the task of predicting media memorability in aide to the Benchmarking Initiative for Multimedia Evaluation's list of tasks for the MediaEval 2022 Workshop. Pre-defined ensemble models are versatile enough to incorporate other pre-defined regressors and ensemble models as base estimators and build on the knowledge accumulated by them. Video-level features like the C3D were chosen, and the regressors and ensemblers were trained on these features with parameter optimization.
The task of assigning a score for video memorability can be formulated as a regression problem.
In recent years, there have been numerous advances in the study of regression and classification
on video in academic literature. The value of combining features from diferent modalities has
been thoroughly demonstrated in numerous earlier works [
By building on top of the previous editions, we approached this task as an optimisation
challenge. Seminal papers on Ensemble learning [
The modalities and traits that are best predictive of memorability remain unknown. In light of this, we decided to use Ensembler-based methods in our quest to achieve a breakthrough.
In this proposed work to measure the video memorability, we opted to use 3D Convolution
(C3D) features [
The models were trained using an 80-20 training-validation split based on Video ID. To test the efectiveness of the various models, we ran each model individually on the dataset and carefully handpicked those that gave the best metrics. The metrics were based on Mean Absolute Error (MAE), Mean Squared Error (MSE), Root Mean Squared Error (RMSE) and Spearman rank correlation coeficient. Ensemble learning tends to perform better when there is a diversity among the models used and this was an additional contribution to our choice of models.
Previous work[
RMSE and Spearman’s were the deciding factors based on which we chose the methods after rigorous experimentation. Internal tests were done on the 80/20 training/validation split.
Models that are efective in predicting continuous output variables, memorability scores in this case, were chosen.
The models that achieved low RMSE scores and high Spearman’s rank correlation coeficient ( ) on the training set were ascertained and are added to a regressor candidate pool which was initially kept empty. Ensemble models, such as the Gradient Boosting Regressor, AdaBoost Regressor, and XGBoost Regressor, do not require additional base estimators and can be trained independently. Therefore, they were also added to the regressor candidate pool. On the other hand, Voting Regressor and Stacking Regressor permit the use of pre-trained base estimators. The regressors in the candidate pool with the best performance, which can be observed from Table 1, were chosen as the base estimators for Voting and Stacking Regressors. Since Voting Regressor allows 3 base estimators, 4 best models based on performance from the pool were chosen and 4 instances of Voting Regressor were made with diferent permutations of the 3 base estimators. The best performance was observed in the model with AdaBoost, Gradient Boosting, and Bayes Regression as best estimators. Similarly, 4 instances of Stacking Regressor models were made, whose 3 base estimators were permuted with 4 of the best performing models in the pool which now includes Voting Regressor. The best performance was observed in the model with Voting Regressor, AdaBoost Regressor, and Gradient Boosting Regressor as base estimators.
Six ensemble methods were chosen initially of which two made the final cut. One of the base
estimators in the proposed architecture for the final Ensemblers (Voting and Stacking) is the
Gradient Boosting Regressor, which serves as an Ensembler itself. max_depth is a parameter
that was identified as an interesting value to tweak [
The performance of the proposed architecture was evaluated using the metrics namely Pearson correlation coeficient, Spearman’s rank correlation coeficient, and MSE.
From the results observed on the training set reported in table 1, have given a Spearman coeficient of around 0.52. This indicates a positive and above average correlation between the two metrics. Out of all the regressors in the candidate pool, Voting Regressor and Stacking Regressor performed the best with Spearman’s coeficient of 0.507 and 0.520 respectively. The MSE values of both ensemblers were 0.008. The other regressors gave around 0.45 Spearman’s coeficient but the best two were picked namely, Stacking Ensembler and Voting Ensembler, and submitted for evaluation. The training set’s result of the final ensemblers are highlighted in table 1.
The submitted runs of the Stacking and Voting ensemblers were evaluated and the results are tabulated in table 2. The results using the training data and testing data show a low variance in the performance implying the model is not over-fitted on the training data. The Spearman’s coeficient for the Stacking Ensembler and Voting Ensembler are 0.525 and 0.513 respectively which means an above average correlation between our predicted scores and the ground truth values. The Pearson coeficient also indicates the same. The most notable metric from submitted runs is the MSE of 0.008 for both Ensemblers.
In summary, the results indicate that Stacking regressor was able to perform better than the others. Although it was found not to be very diferent from Voting Regressor, Stacking Regressor achieved the best correlation and MSE scores.
In this competition, we’ve built a strong upper layer based on the foundation and precedents established by previous work. Our key takeaway lies in aiding to establish ensemble learning as a great way to approach the challenge of predicting media memorability.
We emphasise that ensemble models from multiple methods produces the best results. Future work would ideally experiment further with the parameters, diferent base estimators, and diferent ensemble techniques other than the ones mentioned in our approach.