The modeling of video memorability is still an open challenge for researchers in machine learning. This paper presents our methodology for the MediaEval 2022: Predicting Media Memorability Challenge. The proposed approach investigated ensemble learning methods using the pre-extracted image features: Alexnet, Resnet, and Densenet. In addition to that Transformers and TF-IDF modeling were done on text features. Further, image and text features were ensembled using late fusion that helped in predicting the video memorability on the Memento10k dataset with an accuracy of 0.661.
The study [
The studies [
The models were applied to the textual features provided as the video description in the Memento10k dataset. Several machine learning techniques are applied based on text inputs as well as visual information. The video frames were modeled and only the best-performed models were used to predict the media memorability. The multi-modal approach is applied to encode textual and visual features. These embeddings are provided to linear regression models to predict the best media memorability.
The pre-extracted features are provided as image features from video frames of the beginning, middle and last of each video dataset. The machine learning model is applied where the topscoring pre-extracted features are selected. The top resulting features include ALEX-NET, RES-NET, DENSE-NET, EFFICIENT-NET, and VGG. Further analysis is carried out by applying XG Boost, ADA Boost, Random Forest, KNN, and MLP (Multi-layer Perceptron) where ADA Boost outperformed and provide the best results. Figure 1 shows the complete understanding of the approach taken to accurately predict the media memorability.
The video caption or information helps in understanding the context of the video frame using
semantic techniques. These features help in providing a baseline understanding of the
video/image frame. TF-IDF[
Text Transformers provides another approach that helps in understanding the semantics of a
video. There are diferent text transformers that are applied over the provided Memento10k
dataset. The applied transformers help in getting the similarities and dissimilarities among
the sentences which helped in identifying topics in text data. We got the best result using
DistilBERT[
Videos are nothing but a continuous transition of images. For image-related features, 3 frames
of every video were considered i.e. first, last and middle. Our approach consisted of learning the
best pre-extracted feature that is already provided in the Memento10k dataset. After training
on several machine learning models, only the top 3 image features were selected which were
giving best accuracies on validation data. Namely, Alexnet[
The ensemble combines more than one machine learning model that outperforms in predicting more accurately. This technique also provides a concept of a multi-classification model where the resulting factor to predict is based on combining multiple models. After running the base model on diferent features, late fusion was applied. Linear regression and average-out techniques were used in the ensemble step. 3.4.1. Runs Details Run 1: Only image features were considered and then an average was taken of all 3 of them wto visualize the efect of only image features in predicting media memorability. Run 2: Average of Transformers and TF-IDF was taken to see if there was any efect of the image features as captions were the depiction of the same video. Run 3: Image features and TF-IDF scoring were considered and averag was taken. Run 4: All the best models were considered, from the image/video baseline Ada Boost was picked for all the features, and for text, Transformers and TF-IDF both were considerd. And then late fusion was done by taking the average of them, this gave us an improvement on the baseline methods. Run 5: Results from image features together with TF-IDF were passed to a Linear Regression Model.
The results elaborate on Spearman’s correlation coeficient, Pearson’s correlation coeficient, and Mean Square Error (MSE) values over the validation and testing set of data. The results in Table 1 show the obtained correlation against the validation and the results in Table 2 shows the obtained correlation on testing of the dataset of Memento10k for each submitted execution.
It is observed that Spearman’s correlation coeficient started predicting with 0.421 accuracies and suddenly improved in the prediction accuracy in its second execution and till the fifth execution. The maximum accuracy gained from Spearmen’s correlation coeficient is 0.661. On the other side, Pearson’s correlation coeficient provides a little improved accuracy when compared with Spearman’s correlation coeficient. Pearson’s correlation coeficient obtained an accuracy of 0.439 on the test dataset improved quickly in next following executions. The maximum obtained accuracy from Pearson’s correlation coeficient is 0.667. Furthermore, the MSE also reduced from 0.021 to 0.006 during several executions on the testing dataset.
The approach of multi-classification through ensemble provides a better approach to predicting media memorability. The textual features and visual features predict well when modeled through ensemble learning. The textual features provide the semantic context while the visual features provide the understanding of each image through pre-extracted features from the Memento10k dataset. The accuracy to predict the media memorability increased during the testing dataset in its second execution by changing the hyperparameters of the machine learning models. TF-IDF is already a better approach to predict media memorability but our approach showed that TF-IDF performed way better while modeling through Linear Regression.
This work was supported in part by the Higher Education Commission (HEC) Pakistan, and in part by the Ministry of Planning Development and Reforms under the National Center in Big Data and Cloud Computing.