Results of the 2016 ENtity Summarization Evaluation Campaign (ENSEC 2016) Kalpa Gunaratna1 , Gong Cheng2 , Andreas Thalhammer3 , and Qingxia Liu2 1 Kno.e.sis Center, Wright State University, USA 2 National Key Laboratory for Novel Software Technology, Nanjing University, China 3 Institute AIFB, Karlsruhe Institute of Technology, Germany kalpa@knoesis.org,gcheng@nju.edu.cn,andreas.thalhammer@kit.edu, qxliu.nju@gmail.com Abstract. Entities and their descriptions are becoming an important part of the datasets and knowledge graphs available on the Web. These descriptions can be used in concise representation (i.e., summaries) to help users understand the Web content (e.g., summaries generated from Google Knowledge Graph in Google Search). In the recent past, several systems emerged to tackle the problem of automatic summary genera- tion for entity descriptions. Even though these proposed systems contin- uously push the boundaries, the problem is not yet resolved completely. Therefore, there is a need to support and encourage researchers in the community to participate in solving this important problem. ENSEC, the entity summarization evaluation campaign, is the first step taken towards realizing that goal, and we present the results of the systems participating in the campaign. 1 Introduction The volume of entity-centric data is rapidly increasing on the Web, including RDF-based Linked Data, Schema.org annotations, Facebook’s Open Graph, and Google’s Knowledge Graph. The interlinked datasets, entity annotations, and knowledge graphs on the Web describe entities (e.g., actors and films) and rela- tions between them (e.g., starring). The description of an entity, consisting of a set of entity-property-value triples, is often too long to present to a user, primar- ily for (quick) understanding of the entity. As a substitute, a summary generated from the entity description can be used to efficiently support an end-user task (e.g., browsing, searching). Specifically, an entity summary is a subset of entity-property-value triples selected from the description of an entity. Entity summarization is the process of automatically generating a high-quality entity summary, to be used for a specific task or general purpose. Several systems and approaches have been pro- posed [1, 2, 4] in the recent past in tackling this problem but we believe it is still far from being solved. Therefore, this ENtity Summarization Evaluation Cam- paign (ENSEC) is initiated and organized for the first time to assess strengths 2 K. Gunaratna, G. Cheng, A. Thalhammer, and Q. Liu and weaknesses of entity summarization systems, compare performance of tech- niques, and enhance communication among researchers and developers. In the coming years, we intend to strengthen the gold standards and create benchmarks as new requirements appear. ENSEC 2016 consists of two tracks: the DBpedia-50 track and the LinkedMDB- 30 track, and we invited researchers to try out their new proposals. DBpedia-50 and LinkedMDB-30 tracks consist of 50 and 30 entities, respectively. A system had the opportunity to participate in both or either of the tracks. Each system is evaluated against newly created gold-standard summaries by human judges. The remainder of this report is organized as follows. In Section 2, we describe the two tracks. In Section 3, we characterize gold-standard entity summaries given by human judges. In Section 4, we present evaluation metrics. In Section 5, we report evaluation results. Finally, in Section 6, we conclude this report. 2 Tracks ENSEC 2016 consists of two tracks: the DBpedia-50 track and the LinkedMDB- 30 track. 2.1 The DBpedia-50 Track The DBpedia-50 Track consists of 50 entities in DBpedia (version 2015-04)4 . Their descriptions are restricted to the following datasets: – core data: DBpedia Ontology, Mapping-based Types, Mapping-based Prop- erties, Titles, – domain-specific data: Images, Geographic Coordinates, Homepages, Person- data, – categories: Article Categories, Categories (Labels), Categories (Skos), and – YAGO types: YAGO types, YAGO type hierarchy. For diversity purposes, the 50 entities are composed of ten entities randomly selected from each of the following five classes in DBpedia: Agent, Place, Work, Species, and MeanOfTransportation. The description of each entity consists of at least twenty RDF triples; an entity can be either the subject or the object of a triple. All the 50 entity descriptions can be found online5 . Each participating system should select a subset of five triples from the de- scription of each entity, as a summary for general purposes. Considering that some systems can be configured in different ways (e.g., under different parame- ter settings), each participating system is allowed to submit the results of two runs under different configurations. 4 http://wiki.dbpedia.org/Downloads2015-04 5 http://km.aifb.kit.edu/ws/sumpre2016/dbpedia50.zip Results of ENSEC 2016 3 2.2 The LinkedMDB-30 Track This track consists of 30 entities in LinkedMDB (version 2012-02-10)6 . For diversity purposes, the 30 entities are composed of ten entities randomly selected from each of the following three classes in LinkedMDB: Film, Actor, and Director. The description of each entity consists of at least twenty RDF triples; an entity can be either the subject or the object of a triple. All the 30 entity descriptions can be found online7 . Each participating system should select a subset of five triples from the de- scription of each entity, as a summary for general purposes. Considering that some systems can be configured in different ways (e.g., under different parame- ter settings), each participating system is allowed to submit the results of two runs under different configurations. 3 Gold-standard Entity Summaries Entity summaries generated by participating systems are compared against the gold-standard entity summaries created by a group of students at Karlsruhe In- stitute of Technology, Nanjing University, and Wright State University, as human judges. In the rest of the report, we refer to these gold-standard summaries as ideal summaries. We asked 10 independent human evaluators to generate ideal summaries of length 5 for each entity. That is, each ideal summary consists of five triples selected from the description of an entity for general purpose (not task-specific summaries). Participating systems in the ENSEC 2016 campaign showed less interest in the DBpedia-50 track and hence we focused on completing ideal summaries for the LinkedMDB-30 track8 . For the LinkedMDB-30 track, each entity received at least 6 different ideal summaries from 6 different independent evaluators. These ideal summaries and the summaries generated by the participating systems are available online9 . In the following evaluation and results section, we present results for only the LinkedMDB-30 track. 4 Evaluation Metrics We use the evaluation metrics as presented in Equations 1 and 2. When there are n ideal summaries denoted by SummIi (e) for i = 1, .., n and an automati- cally generated summary denoted by Summ(e) for entity e, the agreement on ideal summaries is measured by Equation 1 and the quality of the automatically generated summary is measured by Equation 2. In other words, the quality of 6 http://www.cs.toronto.edu/~oktie/linkedmdb/linkedmdb-latest-dump.zip 7 http://km.aifb.kit.edu/ws/sumpre2016/linkedmdb30.zip 8 We will complete and strengthen the DBpedia-50 track in future. Currently, it con- sists of at least two different ideal summaries per entity. 9 http://km.aifb.kit.edu/ws/sumpre2016/ENSEC2016_LinkedMDB.zip 4 K. Gunaratna, G. Cheng, A. Thalhammer, and Q. Liu an entity summary is its average overlap with the ideal summaries for the entity in the gold standard. n n 2 X X Agreement(e) = |SummIi (e) ∩ SummIj (e)| (1) n(n − 1) i=1 j=i+1 n 1X Quality(Summ(e)) = |Summ(e) ∩ SummIi (e)| (2) n i=1 If we are considering only outgoing triples of an entity as its description, we simply compare the property-value pairs for agreement and summary quality. Because we are considering both incoming and outgoing triples for each entity, we compare the whole triple in computing the values using Equations 1 and 2. System Average Summary Quality System 1-A 1.9722 System 1-B 1.9388 System CD 0.6444 Average Agreement 1.8199 Table 1: Average summary quality and average agreement (of ideal summaries) for the LinkedMDB-30 track for summary length of five. 5 Results Table 1 presents the evaluation results of the systems participating in this cam- paign for LinkedMDB-30 track. System 1 [3] has two variations that we named System 1-A and System 1-B, received from the East China University of Science and Technology. The submission received from Nanjing University is named as System CD [5]. These system descriptions are included in the SumPre 2016 workshop proceedings. Table 2 presents summary quality results running each system for each entity in the LinkedMD-30 track. 6 Conclusion and Remarks ENSEC 2016 consists of two entity samples taken from DBpedia and Linked- MDB datasets. We created a gold standard for LinkedMDB entity sample and evaluated the participating systems. We hope to continue this campaign together with the SumPre workshop series to support the community in creating better entity summaries that can lead to improvements in real-world practical systems. Further, we will investigate on new evaluation metrics to measure the quality of the entity summaries in future, which can complement the existing measures proposed in the literature. Results of ENSEC 2016 5 Entity Type Entity ID System 1-A System 1-B System CD 84 1.6666 1.6666 0.6666 766 1.5 1.5 0.5 30242 2.3333 2.3333 0.3333 30510 2 2 0.6666 33739 2.8333 2.8333 0.6666 Actor 35586 2.1666 2.1666 1 36732 2.3333 2.3333 0.5 39131 1.6666 1.6666 0.6666 41033 1.3333 1.3333 0.6666 43934 2.6666 2.6666 0.8333 261 1.6666 1.6666 0.6666 8420 1.3333 1.3333 0.6666 8532 1.3333 1.3333 0.6666 8685 1.6666 1.6666 0.6666 9424 1.5 1.5 0.6666 Director 9562 2.3333 2.3333 0.5 10576 2.3333 2.3333 0.3333 11372 1.3333 1.3333 0.3333 11556 2.1666 2.1666 0.6666 20668 2.1666 2.1666 0.8333 1511 1 1 0.1666 7751 1.8333 1.8333 0.8333 38753 2.3333 1.3333 0.6666 40691 2.3333 2.3333 0.1666 45556 2.3333 2.3333 0.1666 Film 49486 2.3333 2.3333 0.6666 55491 1.5 1.5 0.5 66221 2.1666 2.1666 1.5 68798 2.3333 2.3333 1.1666 81553 2.6666 2.6666 1 Table 2: Summary quality results of each system for each entity in the LinkedMDB-30 track. Entity ID reflects the actual ID in the LinkedMDB database. 6 K. Gunaratna, G. Cheng, A. Thalhammer, and Q. Liu Acknowledgments. We thank all the participants in the creation of gold- standard entity summaries. Gong Cheng and Qingxia Liu were supported in part by the NSFC under Grant 61572247 and 61223003, and in part by the Fundamental Research Funds for the Central Universities. Andreas Thalham- mer was supported by the German Federal Ministry of Education and Research (BMBF) within the Software Campus project “SumOn” (grant no. 01IS12051). Kalpa Gunaratna received partial support from the National Science Founda- tion (NSF) award: EAR 1520870: Hazards SEES: Social and Physical Sensing Enabled Decision Support for Disaster Management and Response. References 1. Cheng, G., Tran, T., Qu, Y.: RELIN: Relatedness and Informativeness-based Cen- trality for Entity Summarization. In: Aroyo, L., Welty, C., Alani, H., Taylor, J., Bernstein, A., Kagal, L., Noy, N., Blomqvist, E. (eds.) ISWC 2011, Part I. LNCS, vol. 7031, pp. 114–129. Springer, Berlin Heidelberg (2011) 2. Gunaratna, K., Thirunarayan, K., Sheth, A.: Diversity-aware Entity Summariza- tion Using Incremental Hierarchical Conceptual Clustering. In: 29th AAAI Con- ference on Artificial Intelligence, pp. 116–122 (2015) 3. Li, Y., Zhao, L.: A Common Property and Special Property Entity Summarization Approach Based on Statistical Distribution. In: 2nd International Workshop on Summarizing and Presenting Entities and Ontologies (2016) 4. Thalhammer, A., Toma, I., Roa-Valverde, A.J., Fensel, D.: Leveraging Usage Data for Linked Data Movie Entity Summarization. In: 2nd International Workshop on Usage Analysis and the Web of Data (2012) 5. Xu, D., Zheng, L., Qu, Y.: CD at ENSEC 2016: Generating Characteristic and Diverse Entity Summaries. In: 2nd International Workshop on Summarizing and Presenting Entities and Ontologies (2016)