Entities and their descriptions are becoming an important part of the datasets and knowledge graphs available on the Web. These descriptions can be used in concise representation (i.e., summaries) to help users understand the Web content (e.g., summaries generated from Google Knowledge Graph in Google Search). In the recent past, several systems emerged to tackle the problem of automatic summary generation for entity descriptions. Even though these proposed systems continuously push the boundaries, the problem is not yet resolved completely. Therefore, there is a need to support and encourage researchers in the community to participate in solving this important problem. ENSEC, the entity summarization evaluation campaign, is the rst step taken towards realizing that goal, and we present the results of the systems participating in the campaign.
The volume of entity-centric data is rapidly increasing on the Web, including RDF-based Linked Data, Schema.org annotations, Facebook's Open Graph, and Google's Knowledge Graph. The interlinked datasets, entity annotations, and knowledge graphs on the Web describe entities (e.g., actors and lms) and relations between them (e.g., starring). The description of an entity, consisting of a set of entity-property-value triples, is often too long to present to a user, primarily for (quick) understanding of the entity. As a substitute, a summary generated from the entity description can be used to e ciently support an end-user task (e.g., browsing, searching).
Speci cally, an entity summary is a subset of entity-property-value triples
selected from the description of an entity. Entity summarization is the process
of automatically generating a high-quality entity summary, to be used for a
speci c task or general purpose. Several systems and approaches have been
proposed [
ENSEC 2016 consists of two tracks: the DBpedia-50 track and the LinkedMDB30 track, and we invited researchers to try out their new proposals. DBpedia-50 and LinkedMDB-30 tracks consist of 50 and 30 entities, respectively. A system had the opportunity to participate in both or either of the tracks. Each system is evaluated against newly created gold-standard summaries by human judges.
The remainder of this report is organized as follows. In Section 2, we describe the two tracks. In Section 3, we characterize gold-standard entity summaries given by human judges. In Section 4, we present evaluation metrics. In Section 5, we report evaluation results. Finally, in Section 6, we conclude this report. 2
ENSEC 2016 consists of two tracks: the DBpedia-50 track and the LinkedMDB30 track. 2.1
The DBpedia-50 Track consists of 50 entities in DBpedia (version 2015-04)4. Their descriptions are restricted to the following datasets: { core data: DBpedia Ontology, Mapping-based Types, Mapping-based Properties, Titles, { domain-speci c data: Images, Geographic Coordinates, Homepages, Persondata, { categories: Article Categories, Categories (Labels), Categories (Skos), and { YAGO types: YAGO types, YAGO type hierarchy.
For diversity purposes, the 50 entities are composed of ten entities randomly selected from each of the following ve classes in DBpedia: Agent, Place, Work, Species, and MeanOfTransportation.
The description of each entity consists of at least twenty RDF triples; an entity can be either the subject or the object of a triple. All the 50 entity descriptions can be found online5.
Each participating system should select a subset of ve triples from the description of each entity, as a summary for general purposes. Considering that some systems can be con gured in di erent ways (e.g., under di erent parameter settings), each participating system is allowed to submit the results of two runs under di erent con gurations. 4 http://wiki.dbpedia.org/Downloads2015-04 5 http://km.aifb.kit.edu/ws/sumpre2016/dbpedia50.zip
This track consists of 30 entities in LinkedMDB (version 2012-02-10)6.
For diversity purposes, the 30 entities are composed of ten entities randomly selected from each of the following three classes in LinkedMDB: Film, Actor, and Director.
The description of each entity consists of at least twenty RDF triples; an entity can be either the subject or the object of a triple. All the 30 entity descriptions can be found online7.
Each participating system should select a subset of ve triples from the description of each entity, as a summary for general purposes. Considering that some systems can be con gured in di erent ways (e.g., under di erent parameter settings), each participating system is allowed to submit the results of two runs under di erent con gurations. 3
Entity summaries generated by participating systems are compared against the gold-standard entity summaries created by a group of students at Karlsruhe Institute of Technology, Nanjing University, and Wright State University, as human judges. In the rest of the report, we refer to these gold-standard summaries as ideal summaries. We asked 10 independent human evaluators to generate ideal summaries of length 5 for each entity. That is, each ideal summary consists of ve triples selected from the description of an entity for general purpose (not task-speci c summaries).
Participating systems in the ENSEC 2016 campaign showed less interest in the DBpedia-50 track and hence we focused on completing ideal summaries for the LinkedMDB-30 track8. For the LinkedMDB-30 track, each entity received at least 6 di erent ideal summaries from 6 di erent independent evaluators. These ideal summaries and the summaries generated by the participating systems are available online9. In the following evaluation and results section, we present results for only the LinkedMDB-30 track. 4
We use the evaluation metrics as presented in Equations 1 and 2. When there are n ideal summaries denoted by SummiI (e) for i = 1; ::; n and an automatically generated summary denoted by Summ(e) for entity e, the agreement on ideal summaries is measured by Equation 1 and the quality of the automatically generated summary is measured by Equation 2. In other words, the quality of 6 http://www.cs.toronto.edu/~oktie/linkedmdb/linkedmdb-latest-dump.zip 7 http://km.aifb.kit.edu/ws/sumpre2016/linkedmdb30.zip 8 We will complete and strengthen the DBpedia-50 track in future. Currently, it consists of at least two di erent ideal summaries per entity. 9 http://km.aifb.kit.edu/ws/sumpre2016/ENSEC2016_LinkedMDB.zip an entity summary is its average overlap with the ideal summaries for the entity in the gold standard.
Agreement(e) =
2 n(n 1) n n X X jSummiI (e) \ SummjI (e)j i=1 j=i+1
n Quality(Summ(e)) = 1 X n i=1 jSumm(e) \ SummiI (e)j (1) (2)
If we are considering only outgoing triples of an entity as its description, we simply compare the property-value pairs for agreement and summary quality. Because we are considering both incoming and outgoing triples for each entity, we compare the whole triple in computing the values using Equations 1 and 2. ENSEC 2016 consists of two entity samples taken from DBpedia and LinkedMDB datasets. We created a gold standard for LinkedMDB entity sample and evaluated the participating systems. We hope to continue this campaign together with the SumPre workshop series to support the community in creating better entity summaries that can lead to improvements in real-world practical systems. Further, we will investigate on new evaluation metrics to measure the quality of the entity summaries in future, which can complement the existing measures proposed in the literature. Acknowledgments. We thank all the participants in the creation of goldstandard entity summaries. Gong Cheng and Qingxia Liu were supported in part by the NSFC under Grant 61572247 and 61223003, and in part by the Fundamental Research Funds for the Central Universities. Andreas Thalhammer was supported by the German Federal Ministry of Education and Research (BMBF) within the Software Campus project \SumOn" (grant no. 01IS12051). Kalpa Gunaratna received partial support from the National Science Foundation (NSF) award: EAR 1520870: Hazards SEES: Social and Physical Sensing Enabled Decision Support for Disaster Management and Response.