Multi-modal knowledge graphs (MMKGs) [1,2], which ground non-symbolic data into symbolic entities, have attracted attention as datasets for semantic and conceptual processing across modalities. However, constructing and publishing multi-modal temporal knowledge graphs (MMTKG) that ground multi-modal and time-series data, such as videos, into entities in the graph is still in the early stages.
Typical MMKGs describe multi-modal contents by URLs or file paths. This approach may not be suitable for the permanent publication of MMKGs as the multi-modal contents may become inaccessible due to broken links. This issue could potentially be resolved by encoding the file's binary data as an entity in the KG [3,4]. However, building an MMTKG that describes the content of a video in fine-grained time intervals, such as in seconds or video frames, would result in huge data size, making it expensive to publish and share.
We proposed methods compressing MMTKGs of videos and conducted experiments to determine their effectiveness. We focused on two types of MMTKGs: KGs with video frame images encoded in Base64 and KGs with entire video files encoded in Base64. The proposed methods include differential compression based on knowledge representation of splitting video frame images and reduction of redundant triples based on inference rules. The results demonstrated that our compression methods reduced the size of the MMTKGs by 27.7-36.1%. This study contributes to reducing the cost of distributing large MMTKGs on the web.
Zhu et al. [1] and Chen et al. [2] comprehensively surveyed and summarized works on MMKGs. Typical multimodal knowledge graphs are MMpedia [5] and IMGpedia [6], which ground images to entities in the graph. VisionKG [7] is an MMKG containing bounding boxes (bboxes) of objects extracted from various image datasets such as MS-COCO [8], CIFAR [9], and PASCAL VOC [10]. These MMKGs represent images by URIs or file paths. Studies on video KGs have evolved in the context of video indexing and retrieval [11,12,13]. VEKG [14] is an MMKG based on the extracted events from videos, bboxes, and image features. However, the data is not publicly available. There have been a lot of studies of compression methods for KGs [15]. However, MMKGs for videos are not covered.
MMKGs usually describe images and videos by URIs or file paths, which causes broken links to multi-modal files. Thus, we focus on permanently accessible MMTKGs that embed multi-modal files in a KG as an entity, and propose compression methods for these MMTKGs.
As an example, we constructed MMTKGs of indoor daily activities from multi-modal data of videos, text, and JSON output by VirtualHome-AIST1 [16], as shown in the upper left of Figure 1. The multi-modal data was output every five frames. The dataset contains over 3,500 videos, which include both fixed camera views and third-person views of the camera moving. The average video length is 64.2 seconds, with a maximum of 268.9 seconds and a minimum of 12.5 seconds. We prepared two types of MMTKGs: a KG with every five video frame images encoded in Base64 described as literal values (i.e., image-embedded MMTKG), and a KG with videos encoded in Base64 described as literal values (i.e., video-embedded MMTKG). We reused the Multimedia Semantic Sensor Network (MSSN) ontology [17] and VirtualHome2KG [18] ontology for schema design.
If the MMTKG contains video frame image data, each video frame image is first compressed as a JPEG. Next, each image is split into a grid. The grid image is encoded in Base64 and described in the knowledge representation as shown in the upper right of Figure 1. Here, if there is no difference between the grid image of the current frame and the grid image at the same position in the previous frame, the entity and the literal value of the current grid image are not created, and those of the previous frame are reused.
We adopted MPEG-4 [19] to reduce the video data size. Each video frame entity has a frame number instead of having a Base64 value, and the video entity has a Base64 value for the compressed video. It is possible to extract arbitrary frame images from the video using FFmpeg [20]. The MMTKG size can be further reduced, but long videos take a longer time to decompress.
The MMTKGs have redundant triples if the 2D bboxes are not changed. We reduced the number of entities and triples by referring to the previous entities if the current 2D bboxes have not changed since the previous frame.
Moreover, inspired by the approach of removing triples that can be inferred from the rules [21], we create only the relation equivalentFrame(π ππ , π ππ ) between previous frame entity π ππ and current frame entity π ππ when all 2D bboxes are not changed from the previous frame. We defined the rule as follows: hasMediaDescriptor(π ππ , π ππππ₯ ) β§ equivalentFrame(π ππ , π ππ ) β β hasMediaDescriptor(π ππ , π ππππ₯ ). Similarly, for grid images, we removed triples that can be inferred from the following rule: image(π ππ , π πππππ ) β§ equivalentImage(π ππ , π ππ ) β β image(π ππ , π πππππ ). Note that the image property here refers to a split image.
Tables 1 and 2 show the results of the compression experiments. Our methods achieved data size reductions of 36.1% for image-embedded MMTKG and 28.3% for video-embedded MMTKG.
There is a trade-off between the number of grid divisions and the number of triples. The best strategy is 4 Γ 4. In this study, we experimented with π Γ π grid divisions; however, experiments with πΓπ grid divisions are also necessary for a more detailed analysis. We published MMTKGs in a permanently accessible format. 2 In addition, tools for decoding and extracting images and videos from compressed MMTKG are available.3
We proposed compression methods for two types of MMTKGs: image-embedded and videoembedded MMTKGs. The former MMTKGs can display arbitrary images on the web using HTML <img> tags without decoding the videos. The latter MMTKGs can apply video compression methods, and if the video is decoded, any frame can be extracted based on the frame number of the image. These MMTKGs can help create benchmark datasets for vision-language models since it is possible to extract arbitrary text and images using SPARQL queries [16]. The compression method for image-embedded MMTKGs might be effective for image stream data in which no video file is created. In contrast, the compression method for video-embedded MMTKGs is more effective when video files are available. Our compression methods for MMTKGs are effective for fixed-camera view videos but are less effective for first-person view videos.
We proposed compression methods for two types of permanently available MMTKGs in which video data are directly embedded as literal values. As a result, our methods achieved data size reductions of 36.1% for image-embedded MMTKG and 28.3% for video-embedded MMTKG. The two MMTKG datasets and the tools are available on GitHub. In the future, we will consider combining our methods with other RDF compression methods [22,23].
This paper is based on results obtained from a project, JPNP20006, commissioned by the New Energy and Industrial Technology Development Organization (NEDO), and JSPS KAKENHI Grant Number JP22K18008 and JP23H03688.