=Paper=
{{Paper
|id=Vol-3304/paper18
|storemode=property
|title=Scoring Prediction Model Based on Fusion of Text and Temporal Features
|pdfUrl=https://ceur-ws.org/Vol-3304/paper18.pdf
|volume=Vol-3304
|authors=Haoqian Li,Xuesong Su,Wenguang Zheng,Jiayi Song,Yingyuan Xiao
}}
==Scoring Prediction Model Based on Fusion of Text and Temporal Features==
https://ceur-ws.org/Vol-3304/paper18.pdf
Scoring Prediction Model Based on Fusion of Text and Temporal
Features
Haoqian Li1, Xuesong Su2, Wenguang Zheng1*, Jiayi Song1 and Yingyuan Xiao1
1
School of Computer Science and Engineering, Tianjin University of Technology, Tianjin, China
2
Technology Inspection Center of Shengli Oilfield SINOPEC Dongying, Shandong, China
Abstract
Nowadays, the Internet has penetrated into every household and is inseparable from our lives.
The Internet provides users with a wide range of online products and services, the utilization of
online platforms is much higher than before, and users gradually become dependent on online
consumption. To promote the development of e-commerce ecosystem, the main task is to in-
crease the purchase rate of goods. In order to reduce information overload and meet the diverse
needs of users, a personalized recommendation system that implements recommendation tasks
using review text is an effective solution to increase the purchase rate of users, and it also
solves the problems of sparse data and cold starts. Currently, many deep recommendation
models based on review text have emerged in this field, but these models have some other
problems, such as lack of fusion of multiple features and lack of judgment of feature im-
portance levels.
Keywords:
review text; recommendation system; feature fusion; rating prediction.
1. Introduction
With the development of the Internet, people are now entering the era of big data, and the intelligent
society makes our information exist in every registered end of the application; when we log in our
account to browse the website, this website also records our browsing information, when we shop
online, the information of the purchased products and when we give feedback, our comments on the
products are also recorded one by one. In general, a growing number of studies argue that since reviews
explain users' opinions, they should help to infer potential dimensions for predicting ratings or pur-
chases. The schemes incorporating reviews have since evolved from simple regularization methods to
achieve rating prediction to neural network methods to achieve rating prediction.1
One of the main directions of research in recommender systems is to improve prediction by using
latent information features, especially in cold-start environments where interaction data may be sparse
or noisy. It has been shown in current rating prediction algorithms that introducing review text data into
the rating prediction task can greatly improve the accuracy of recommendations. To enhance the fusion
of features and the importance of focusing on interaction features to make improvements in the accu-
racy of user-item rating prediction, this paper proposes a multi-feature fusion-based rating prediction
model (2TFRS). Our proposed algorithm models the rating prediction problem as a text matching
problem using user-item review text fused with temporal features. The interaction between features is
enhanced by learning methods that capture important matches between user text and item text; secondly,
the model still combines review timing information, user and item rating information. Finally, the
prediction scores are derived through a regression layer.
ICBASE2022@3rd International Conference on Big Data & Artificial Intelligence & Software Engineering, October 21-23, 2022, Guang-
zhou, China
EMAIL: 1369227010@qq.com (Haoqian Li); wenguangz@tjut.edu.cn (Wenguang Zheng)
ORCID: 0000-0002-7604-9893 (Haoqian Li)
©️ 2022 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
145
2. Related works
In this section, we introduce the classical recommendation models incorporating deep learning
techniques in recent years and the innovative points and construction ideas of our proposed model.
In 2016, the word2vec [1] and Item2vec algorithms, borrowed from the field of natural language
processing, learned the user's behavior sequences as sentences for representation. In the same year,
Google released YoutubeDNN [2] which introduced the classical recommendation system architecture,
divided into two phases of recall and ranking, providing ideas for subsequent industrial-grade recom-
mendation systems. Many other classical models were proposed in the same period, such as ConvMF [3],
PNN [4], DNN [5], DeepCrossing [6] etc. Many scholars have found that the accuracy of the prediction
results is greatly improved by taking the attention mechanism into account when modeling the scoring
prediction problem. Seo et al. proposed an attention-based convolutional neural network model
（D-Attn model[7]）to extract potential representations of users and items. In 2017, L Zheng et al.
proposed the DeepCoNN[8] model to mine the nature of user-behavioral goods from user-item review
texts; In 2018, C. Chen et al. proposed the NARRE [9] model, which adds the attention mechanism to
assign weight to each comment based on the DeepCoNN model; in 2020, Parisa Abolfath and Saeedeh
Momtazi proposed the MPRS [10] model, which aims to focus on user -item interaction features; In 2022,
Peilin Yang et al. proposed a deep learning-based main auxiliary network—MAN[11], which helps the
main network to generate rating prediction values by focusing on the deep meaning at the word vector
level with the help of the auxiliary network; this paper is based on this model and presents new ideas.
Considering the influence of time decay factor on recommendation results, combining the temporal
information of reviews as the weights of embedded vectors and then doing local interactions between
vector pairs, performing convolutional operations and then inputting into the regression layer to obtain
predicted scores.
3. Our proposed model
First, we consider each data record as a tuple（u, v, , , ）to represent the review records
written by user u for item v, including the rating , the review and comment time . To con-
struct the user-item matching matrix, a review document-based approach is used. Firstly, all the reviews
written by user u are concatenated into a single document, denoted as , consisting of n words.
Similarly, integrate all the comment texts of item v into one document as where m is the length of
the document.
Next, we use the word vector matching matrix for each pair of user u and item v as input to the
CNN architecture, and each element in the matrix represents the similarity of the pth word in the
user text document to the qth word in the item text document . The first layer of our
proposed model is the embedding layer, which serves to map each word in the review text to a
d-dimensional vector. The words of each user review text and the words of each item review text are
trained by a word embedding function that can be used with the pre-trained embeddings from GloVe [12]
used in Wikipedia. The and in Equation 1 represent the word embedding vectors of word
and word respectively, with the user embedding vector denoted as α and the item embedding vector
as β . In this model, we consider that the user's preferences may change with time, we propose a
weighting function that takes into account the time decay, the user's recent review behavior represents
the user's recent preferences, and the item's recent reviews often represent the recent quality of this item,
which can accurately improve the recommendation performance.
= (1)
Where Δt is the time difference between the time of user u commenting on item v and the current
time. As Δt increases, the weight value of the embedding vector becomes smaller, which means that the
146
user's commenting behavior long ago will have less influence on the prediction score. Then, the user
embedding vector with weights added is denoted as ε and the user embedding vector with weights
added is denoted as η.
To better capture the word meanings between each word for matching, we calculate the similarity
between two words by computing the cosine similarity of their word embedding vectors to construct a
user-item matching matrix to achieve a joint representation of user-item pairs, capture the joint se-
mantic information .
Its similarity is calculated as shown in Equation 2:
(2)
The above is the first layer of the architecture of our proposed model. Next, we input the user-item
pair matching matrix constructed above into the convolutional neural (CNN) architecture.
Rating Score
CNN
F(α) F(β )
Embedding Layer
User Review Text Item Review Text
Figure 1 Scoring prediction model based on text and temporal feature fusion
The CNN framework consists of a convolutional layer, a pooling layer, and a fully connected layer,
each convolutional layer uses a filter to generate a feature map ，defined as Equation 3:
(3)
, The symbol is the convolution operator, is the bias term.
We apply a maximum pooling operation to each feature mapping obtained from the convolutional
layer and use the maximum value in each pooling window as one of the features of the corresponding
kernel. By repeating the convolutional and maximum pooling layers, we allow to filter the interference
and extract each feature simultaneously in a certain proportion. The result of the final maximum pooling
layer is passed to a fully connected layer and then to a single neuron layer with a linear activation
function - the regression layer. Concomitantly, the User-Item prediction score is output, and the
calculation formula 4 is as follows:
(4)
Where O is the result of the fully connected layer, W contains the weights of the regression layer, and
is the bias term.
147
The method proposed in this paper is a rating prediction model based on the fusion of textual and
temporal features to achieve rating prediction, and this model is suitable for regression prediction based
on user-item similarity.
4. Experiments
We used the Amazon Review Dataset-5core public dataset to make an evaluation for each experi-
ment. Each dataset was randomly divided into training set, validation set and test set in the ratio of
80:10:10. Classify the comment data set by category, as shown in Table 1:
All comment texts were first processed by the Stanford Core NLP word generator to obtain the sub
words, then their stop words and punctuation marks are removed, and finally t the word embedding
vectors were trained on the Glove. 6B.50d word list used in Wikipedia.In CNN, we set the number of
convolutional layers to 7, and the number of convolution kernels for each layer is 64; Set the hy-
perparameter batch size to 128 and the dropout rate to 0.5.The root mean square error (RMSE) is used to
evaluate the performance of our proposed algorithm. Let N be the total number of data points to be
tested. Then, define the RMSE (5) as
(5)
In order to better test the performance of our proposed models. For this purpose, we tested the
DeepCoNN model, the NARRE model, the MPRS model, and the MAN model on a subset of Amazon
on the latest version. The mean squared error values (RMSE) obtained for our models as well as for the
models of the comparison experiments on five different classes of the Amazon data subset have been
listed Table 1：
Table 1
RMSE results of the compared methods on different datasets
Deep- 2TFRS(relative
Dataset MPRS MAN
CoNN improvement)
Cell Phones and
1,4129 1.0129 1.1022 1.0743(2.79%)
Accessories
Toy and Games 0,9024 0.8959 0.8384 0.8271(1.13%)
Office Products 0.8692 0.7887 0.7727 0.7536(1.91%)
Amazon instant
1.1132 1.1022 0.9786 0.9471(3.15%)
video
Digital Music 0.9006 0.8760 0.8701 0.8502(1.99%)
As shown in the table, the performance of our proposed model is superior to the baseline experi-
ments after testing on five data subsets. It can be seen that our model obtains the largest relative im-
provement (3.15%) on the Amazon dataset AZ-IV (Amazon instant video) category.
5. Conclusion
After drawing on a large number of excellent papers on recommendation systems, we propose a new
model based on feature fusion to achieve rating prediction. The model is modeled separately by using
user reviews and item reviews, embedding the word vectors, processing the temporal information as the
weights of the embedded vectors, and then calculating the similarity of the user-item vector pairs with
weights to form a matching matrix to The CNN architecture and regression network are used to predict
the user's rating value of items.
148
6. References
[1] Mikolov, Tomas, et al. Efficient estimation of word representations in vector space. arXiv preprint
arXiv:1301.3781(2013)
[2] Covington, Paul, Jay Adams, and Emre Sargin. Deep neural networks for youtube recommenda-
tions. Proceeding of the 10th ACM conference on recommender systems. 2016
[3] D. Kim, C. Park, J. Oh, S. Lee, H. Yu, Convolutional matrix factorization for document con-
text-aware recommendation, in: Proceedings of the 10th ACM Conference on Recommender
Systems – RecSys’6, 2016
[4] Qu, Yuanru, et al. Product-based neural networks for user response prediction. 2016 IEEE 16th
International Conference on Data Mining (ICDM)IEEE,2016
[5] Zhang, Weinan, Tianming Du, and Jun Wang. Deep learning over multi-field categorial data.
European conference on information retrieval. Springer, Cham,2017
[6] Shan, Ying, et al. Deep crossing: Web-scale modeling without manually crafted combinatorial
features. Proceedings of the 22nd ACM SIGKDD international conference on knowledge dis-
covery and data mining.ACM.2016
[7] S. Seo, J. Huang, H. Yang, Y. Liu, Representation learning of users and items for review rating
prediction using attention-based convolutional neural network, in: Proceedings of the 3rd Inter-
national Workshop on Machine Learning Methods for Recommender Systems – MLRec, 2017
[8] L. Zheng, V. Noroozi, P.S. Yu, Joint deep modeling of users and items using reviews for rec-
ommendation, in: Proceedings of the 10th ACM International Conference on Web Search and Data
Mining - WSDM ’17,2017, pp. 425–434
[9] C. Chen, M. Zhang, Y. Liu, S. Ma, Neural attentional rating regression with review-level expla-
nations, in: Proceedings of the 2018 World Wide Web Conference, 2018, pp. 1583–1592
[10] Parisa Abolfath, Beygi Dezfouli, Saeedeh Momtazi, Mehdi Dehghan. Deep neural review text
interaction for recommendation systems, Applied Soft Computing Journal 100 (2021) 106985
[11] Yang, Peilin, et al, MAN: Main-auxiliary network with attentive interactions for review-based
recommendation in: Applied Intelligence-2022pp:1—16
[12] J. Pennington, R. Socher, C. D. Manning, GloVe: Global Vectors for Word Representation,
EMNLP (2014)
149