We evaluate the proposed News2Images on a big media data including more-than one million news articles served through a Korean media portal website, NAVER2, in 2014. Experimental results show our method outperforms a baseline method based on word occurrence in terms of both quantitative and qualitative criteria. Moreover, we discuss some future directions for applying News2Images to personalized news recommender systems.
Most news articles consist of a title, a document, and attached
images. Mathematically, a news article x is defined as a triple
x {t, S ,V } , where t, S, and V denote a title, the set of document
sentences, and an image set. V can be an empty set. A title t and a
document sentence s, s S , are represented as a vector of word
features such as occurrence frequency or word embedding. An
image v, v V is also defined as a vector of visual features such
as Scale invariant feature transform (SIFT) [
Many recent studies have reported that the hidden node values
generated from deep learning models such as word embedding
networks and CNNs are very useful for diverse problems
including image classification [
Formally, a word w is represented as a real-valued vector,
w d , where d is the dimension of a word vector. The vector
value of each word is learned from a large corpus by word2vec
[
News Documents Image
Based Contents
Summarization
Function (Similarity &
Diversity) Sentence Vector Set Synthesized
Images k Extracted Sentence Vectors Retrieved Images Data Flow Function Flow
Image Synthesis Function Similarity Function
Retrieved
Titles Generated
Image Features
News Title-Image Database Learned
CNN Model
News2Images is a method of generating image-based contents from a given news document using summarization and text-toimage retrieval. News2Images consists of three parts including key sentence extraction based on the single document summarization, key sentence-related image retrieval by associating images with sentences, and image-based content generation by synthesizing sentences and images. Figure 2 shows the overall framework of News2Images.
Document summarization is a task of automatically generating a
minority of key sentences from an original document, minimizing
loss of the content information [
A news title is the best sentence consistently representing the entire content of the news. ii) A news article consists of at least two sentences and the entire content is built up by composing its sentences’ content.
For precisely summarizing a news document, thus, it is required that a summarized sentence set consists of the sentences not only semantically similar to its title but also covering the entire content with diverse words. We call the former similarity and the latter diversity.
Formally, a document S is defined as a set of its sentences, S {s1, ...,sM } , where M denotes the number of the sentences included in S. The i-th sentence si is represented as a real-valued vector, si d , where d is the vector size, by word2vec and average pooling. Then, document summarization is formulated with Sk* arg max f Sk , S 1 g Sk , S
Sk S arg max f Sk ,t 1 g Sk , S
Sk S where t denotes the title of S, Sk and Sk* are the set of k sentences extracted and an optimal set among Sk . f Sk , S and g Sk , S denote the similarity and the diversity functions, and is the constant for moderating the ratio of two criteria. The similarity f (s, t) between a given sentence s and a news title t is defined as the cosine similarity between two sentence embedding vectors: f (s,t) s t s t .
For calculating the diversity, we partition the sentences of S into multiple subsets using a clustering method. Because a sentence vector implicitly reflects syntactic and semantic information, multiple semantically distinctive subsets are generated by clustering. For the j-th cluster Cj, we calculate the cosine similarity between all the sentences in Cj and the centroid of Cj. Because the cosine similarity can be negative, we consider a negative value as zero. This value is defined as the diversity: g(s,C j ) s c j s c j where cj denotes the centroid vector of Cj.
Finally, k sentences with the largest value defined in (2) are extracted as the summarization set for the given document. Here we set k to three, which means that a news article is summarized into three image-based contents.
The second subtask is to retrieve the images representing semantics similar to the extracted sentences. Because we use the images attached in news articles, the title of a news including an image can be used as a description sentence of the image. (2) (3) (4) Therefore, the semantic similarity of an image to an extracted sentence is calculated by measuring the similarity between the image title vector and the sentence vector.
Formally, when an image feature vector set, V={v1, …, vN}, is given, the images similar to an extracted sentence sˆ are extracted: sˆ t(v) v* arg max f sˆ, t(v) arg max , vV vV | sˆ || t(v) | (5) where t(v) denotes the title of an image v.
Due to the diversity, sentences which are not directly related to the title may be extracted as a core sentence. We assume that a title is “Yuna Kim decided to participate in 2013 world figure skating championship”, and two extracted sentences are “Yuna Kim will take part in the coming world figure skating championship” and “The competition will be held in February.” In this case, the title is not semantically similar to the second sentence. Thus it is difficult to associate the second sentence with Yuna Kim’s images. For overcoming this, we can additionally use the title vector of the news articles given as a query for pooling word vectors into a sentence vector. The use of the news title does not influence the summarization because the title vector is reflected on all the sentence vectors.
Instead of v*, we can generate a new image vector vˆ by averaging the vectors of top K images with the large similarity value. Then, v* is selected as follows: v* arg max f vˆ, v ,
vV vˆi vVK
R(v)
v , R(v) i (6) (7) where vi is the i-th element of v and R(v) denotes a weight function proportional to the similarity rank. An image more similar to vˆ has a larger R(v).
Readability is a main issue of mobile content service. Therefore we generate new image-based contents instead of using the retrieved images for improving the readability and enhancing the users’ interests. An image-based content includes continuous series of synthesized images where the retrieved images and their corresponding sentences are merged. Figure 1 illustrates an example of the image-based contents from a news document.
We evaluate the proposed News2Images on a big media data including over one million Korean news articles, which are provided by a media portal site, NAVER, in 2014. In detail, the word vectors are learned from all the news documents and the CNN models for constructing image features are trained from approximately 220 thousands of news images, which are related to 100 famous entertainers, movie stars, and sports stars. Also, 6,967 news articles are used as the validation set for evaluating the performance. Three key sentences were extracted from a news article including more than three sentences and we used all the sentences in the news consisting of less than three sentences. Then, 20,224 image-based contents were generated from validation news data in total.
We used the word2vec for word embedding and modified
GoogleNet implemented in Caffe for CNN features [
For comparisons, we used a word occurrence vector based on TF/IDF as a baseline in computing the similarity between sentences and titles, instead of a word embedding vector. TF/IDF has been widely used for text mining, and thus we can verify the effects of deep learning-based word features.
Human efforts are still essential for precisely measuring how similar the generated image-based contents are semantically to the news document given as a query. Instead of manual evaluation by |W| denotes the number of concatenated word vectors.
Correct # Accuracy humans, we consider a classification problem as the similarity evaluation. That is, for a given extracted news sentence, we consider that the retrieved image is similar to the sentence when the persons referred in the sentence exist in the image. It is reasonable because this means the method provides diverse images of a movie star for users when a user reads a news about the star.
Table 1 compares the classification accuracy of the baseline and the proposed method. As shown in Table 1, News2Images outperforms the baseline method. This indicates the word embedding features used in News2Images more precisely represent semantics, compared to TF/IDF-based features. Also, we compared the cosine similarity between the titles of the retrieved images and the extracted sentences using their word embedding vectors. The values are averaged on the titles of 20,224 retrieved images. We can find that our method retrieves the images more semantically similar to the extracted sentences.
We compare the accuracies of the generated contents under four parameters including i) the use of news title for pooling word vectors into a sentence vector, ii) the number of retrieved images for an image feature, iii) the weight for proper nouns, and iv) the size of concatenated word vectors. Table 2 presents the accuracy improvement when the title of the summarized news documents is used. We found that the use of the news title dramatically improves the accuracy as 30% compared to the case in which the titles are not used. Interestingly, News2Images not using titles provides the similar performance to the baseline method using titles. Table 3 shows the effects of averaging multiple image features on sentence-to-image retrieval. This indicates that generating a new image feature from multiple image features has no effect on enhancing the performance. To give more weight to proper nouns can improve the quality of the image-based content generation because proper nouns are likely to be a key content of the news. The results in Table 4 support this hypothesis. The number of concatenated word vectors rarely influences the accuracy. We indicate that the information on word sequences is not essential to classify the person in the images from Table 5.
Figure 3 illustrates good and bad examples of image-based contents from news articles. Most of the images are related to the news contents but the sentences including polysemy or too many words are occasionally linked to images not relevant to the sentences. This is caused that one word is represented as only one vector regardless of its meaning. Also, the representation power of pooling-based sentence embedding can be weaken due to the property of average pooling when a sentence consists of too many words.
We proposed a new method for summarizing news articles into image-based contents, News2Images. These image-based contents are useful for providing the news for mobile device users while enhancing the readability and interests. Deep learning-based text and image features used in the proposed method improved the performance as approximately 24% of the classification accuracy and 0.23 of the cosine similarity compared to the TF/IDF baseline method. Our study has an originality in aspect of generating new image contents from news documents even if many studies on summarization or text-to-image retrieval have been reported. This method can be applied to a personalized news recommender system adding user preference information such as subject categories and persons preferred by a user and feedback information into the method. In detail, we can give a weight to words related to subjects or persons preferred by a user when generating sentence vectors. This strategy allows the sentences which the user is likely to feel an interest in to have higher score in summarization and retrieval, thus exposing the photos which the user prefers.
Evaluation should be also improved. Although we evaluate the proposed method with the cosine similarity-based measure and the classification accuracy, it has a limitation for precisely measuring the similarity between the news articles and the image contents generated. It is required to make a ground truth dataset by humans, which not only helps to more precisely evaluate the model performance and can be used as a good dataset for recommendation as well as image-text multimodal learning. Furthermore, we will verify the effects of News2Images on the improvements of the readability through human experiments as future work.
The proposed method can be improved by adding the module of
efficiently learning a common semantic hypothesis represented
with sentences and images using a unified model [