The metadata and content-based information extraction tasks from heterogeneous le sets are pre-processing steps of many Knowledge Graph Construction Pipelines (KGCP). These tasks often take longer than necessary due to the lack of proper tools that integrate several complementary extraction methods and properties to get a rich output set. This paper presents MEL, a Python-based tool that implements a set of methods to extract metadata and content-based information from unstructured information encoded in di erent source document formats. The results are generated as JSON les, which can: (a) optionally be stored in a document store, and (b) easily be mapped to RDF using a variety of tools such as J2RM. MEL supports more than 20 di erent le types, making it a versatile tool that aids pre-processing tasks as part of a KGCP based on comprehensive con gurable settings.
This paper introduces MEL5, a tool that implements a set of methods to extract metadata and content-based information from various le formats as JSON6 objects. For each supported le type, MEL extracts the textual content from the source document and performs speci c pre-processing and data cleaning tasks. Also, it performs basic text analysis tasks (pattern matching and keyword extraction) and generates the results in a machine-readable format (JSON), preparing Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 5 The submission type of this paper is a demonstration that portraits the tool's general functionality. The demo presents: input document set, initial settings, processing, and output set. The input document set is arti cial in order to show various capabilities of the tool for di erent le types and formats. All resources and demo videos are available at https://w3id.org/kgcp/MEL-TNNT 6 https://www.json.org/ the ground for content-based analysis. MEL is integrated with \The NLP7-NER8 Toolkit" (TNNT ), which automates the extraction task of categorised named entities from the MEL results by using diverse state-of-the-art NLP tools and NER models [5]. MEL implements primitives for metadata and content extraction from unstructured data sets of heterogeneous formats, and along with the TNNT results, it provides the groundwork for content-based analysis. MEL and TNNT were developed in conjunction with J2RM [4], to easily map the JSON results to RDF as part of an automated KGCP9.
MEL has comprehensive metadata extraction support of various le types and
formats. In a nutshell: (
NMumaxb.er Number
Avg. 55 24 >50 28 34 45 54 45 | | | | settings. The supported le types are presented in Table 2. The third column shows the theoretical number of attributes that the tool is able to extract per document type, whilst the fourth column shows the average of the extracted attributes from four use case document sets11. OLE 2 le types12 and .docm can only be processed on Windows operating systems. Speci cally for OLE 2 le types, MEL uses the olemeta tool13.
MEL is fully integrated with TNNT as depicted in Figure 1. The set of Pythonbased methods implemented in MEL are generic and can be applied to extract the content and metadata of all supported le types. MEL uses various opensource packages and tools with complementary capabilities to form a \Swiss army knife" of metadata and content-based information extraction from heterogeneous document sets. As part of the \General-Metadata" extraction task, MEL optionally uses the XML14 output from the NLNZ Metadata Extractor tool15, a Java standalone tool that extracts a comprehensive attribute and 11 Each set ranges from 1,174 to 3,334 documents. They are government documents about assessments, endangered species, federal budget, and procurement, all from the \Australian Government Records Interoperability Framework" (AGRIF) [1] project. 12 https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-oleds/ fdc5e702-d09e-4344-a77f-eb079d41f23f 13 http://www.decalage.info/python/oletools/ 14 Extensible Markup Language. 15 http://meta-extractor.sourceforge.net/ property list from dozens of le formats. The MEL general processing model is presented in Figure 2. It is important to note that each le type has its own speci c processing model as well as the text analysis task, which is the last step that is performed for any output. 4
The most comprehensive and current state-of-the-art tool for content extraction and analysis is Apache Tika16, which is a complete and complex Java-based general-purpose system. While MEL's core goals resemble the ones of Apache Tika, the main di erence and bene t of MEL as compared to Apache Tika is that it is a lightweight Python-based package for the metadata extraction of common le formats aimed to be used in a KGCP. Although there is a wide range of Python-based tools and libraries for metadata extraction, to the best of our knowledge, there is no package available that fully integrates in one system a comprehensive set of methods for metadata and content extraction of common le formats that generate the results in JSON structures based on the document 16 https://tika.apache.org/ object model of each format type. Last, MEL can assist in the information extraction stage of several KGCPs, such as the ones described in [6], [2], and [3]. 5
MEL provides a versatile mechanism to extract metadata and content-based information from unstructured data sets of heterogeneous le formats, agnostic of the data sets' domain (general purpose). It has been tested over thousands of documents using di erent formats and datasets as part of the AGRIF project. Based on the structure of the MEL's JSON results, it is possible to easily add a vocabulary or light-weight ontology using JSON-LD17 annotations, in order to make the extracted metadata \RDF ready". This will be explored in the near future leveraging on the integration with JSON-LD ontologies. More le formats will be added in a per use-case requirements basis, in order to support KGCP tasks. Additionally, a project to \containerise" the MEL+TNNT tools is planned in the near future.
The major contributions of this tool are: (