MetaVD is a meta video dataset that interlinks six existing Computer Vision related datasets such as Charades and Kinetics-700. While MetaVD contributes to enhancing video recognition performance, we found two issues as follows. First, some is-a relationships defined and linked by MetaVD are inconsistent, including one circulation of is-a relationships. Second, all concepts in MetaVD are from the six datasets, and therefore some of them are not well semantically arranged, leading to a possibility of ineficient training of video recognition models. Here, we propose a knowledge graph dataset in Resource Description Framework (RDF), which links MetaVD concepts to those in the Commonsense Knowledge Graph (CSKG) and RDFizing MetaVD itself. By linking it, we can more easily detect inconsistent concept relationships. Furthermore, it allows us to link MetaVD concepts to those of conceptually higher ones. Then, some SPARQL queries were issued to it to evaluate its feasibility. The RDF dataset and SPARQL queries mentioned in this extended abstract are downloadable from https://github.com/aistairc/ MetaVD-CSKG .
and Kinetics-700 [
recognition. They defined three relation types of equality, similarity, and hierarchy, and then annotated 568,015 relation labels in total by hand to the pairs of concepts obtained from the original datasets using these relationships. After interlinking these datasets, they confirmed that video recognition performance were increased by proposing two methods of how to utilize the interlink.
relationships. For example, there are the following relationships in MetaVD: Playing_ice_hockey
is-a hit, and hit is-a Volleyball. In addition, there is a circulation for three is-a relationships.
Since the is-a relationship reflects a semantically hierarchical and directional relation between
concepts of the original datasets, a concept in the circulating relationships is claimed to be more
general than itself; that is to say, it is inconsistent. Second, MetaVD interlinks the six datasets,
and all the concepts are from them. Therefore, there are some cases where a more general
concept can be used to group a set of concepts that have a common trait, leading to a possibility
of gaining a better recognition performance [
widely-recognized sources, such as ConceptNet and Visual Genome and we found that almost all MetaVD concepts can be linked with ConceptNet concepts. The conversion of MetaVD into
is required to train a special-purpose human-action recognition model tailored for distinct
2.1. MetaVD to ConceptNet
ifrst attempted to align them. Of the 966 MetaVD concepts, 397 were fully aligned and 560 were partially aligned. "Fully aligned" here means that both concepts exactly match each other after normalization described below. The seven concepts that could not be aligned include
aligned to a MetaVD concept consisting of multiple words such as Turning on a light. In addition, since some aligned concepts were identical words such as accordion for playing_accordion [Kinetics-700] and Playing_accordion [ActivityNet], the total number of concepts is 868.
We used ConceptNet Numberbatch4 to obtain a corresponding ConceptNet ID. The version we obtained is 19.08 English. We extracted headwords from it and constructed an index based on them. Before looking it up, we normalized the terms as follows: • camel case to multiple words • all letters to lowercase • dash symbol to underline • expressing multiple senses using slash symbols expanded to each concept
UCF101 HMDB51 weavsingi_mfabriiclar crocheting Knittingequal knitting STAIR Actions ActivityNet is-a pla KCinheatircasd-7e0s0 hit ying_ice_hockeyis-aLongSbkaotaerBdoinagrding simiBlaandrMaprlacyhiinngg_accordion MetaVD knitting is a plyn aigchce ie
oky is a aordinis a c c o .. .
inte is a lligentagenta action
c is a tivity is a skateboianrsdtirnugm playing et n CSKG
ConceptNet
Roget Visual Genome
WordNet ATOMIC Wikidata FrameNet 2.2. MetaVD to CSKG
that the retrieved CSKG nodes are within one hop from the ConceptNet IDs. Fig 1 delineates relationships between CSKG and MetaVD in our work. We linked MetaVD nodes to CSKG ones.
We used Knowledge Graph Toolkit (KGTK) [
2.3. Degree of abstraction
that ConceptNet graph structure can provide supporting evidence. Each ConceptNet node often has IsA relationships, such as an accordion IsA instrument, and semantically higher concepts have more child nodes linked with the IsA relationship. The number of children of accordion is three, and that of instrument is 68. 2.4. RDFization
TogoDB accepts table data in CSV or TSV formats and provides a GUI-based environment where we can edit configuration that defines how to generate RDF data from a given table. We took a general approach of RDFization from a table dataset. A row becomes a set of triples whose subject is from the cell value of the ID column and other column names and the corresponding cell values denote its properties. RDF dataset from the MetaVD-CSKG data were also built in the same way. The number of triples for MetaVD and MetaVD-CSKG were 25 218 and 1 190 904, respectively. We also built an ancillary RDF dataset to link MetaVD data IDs to ConceptNet IDs including partial words of MetaVD concepts. We loaded the RDF dataset into Fuseki Version 4.7.06.
3.1. Inconsistency Checking
we say inconsistent in terms of the degree of abstraction obtained by counting children of a concept in ConceptNet. For example, while Volleyball has one child and hit has 13, hit is-a Volleyball in MetaVD. 3.2. MetaVD subsetting for customization
tailored to one’s specific purpose. As a feasibility study, we issued a SPARQL query to retrieve all MetaVD concepts related to intelligent agent activity, which returned 200 results. It includes Playing_ice_hockey from ActivityNet, roller_skating from Kinetics-700, and playing_guitar from
addition, we defined degree of abstraction based on the ConceptNet graph structure. By using these data, we propose a way of showing potentially semantically inconsistent relationships in