In this paper, we report on our proposed approach towards a standardized description for systems combining machine learning (ML) components with techniques developed by the Semantic Web (SW) community (SWeMLS), which is one of lessons learned from our large-scale survey (476 papers) on the topic. We elaborate the key information that should be described of SWeMLS and selected methods to support its documentation.
1. Introduction 2. Relevant SWeMLS information In order to adequately represent SWeML Systems, we identify four categories of relevant system information: (i) System Settings, (ii) System Overview, (iii) System Details, and (iv) System Evaluation, which will be briefly described in the following.
System Settings To estimate the applicability of a presented approach, the description of the domain in which a SWeMLS was evaluated is essential. The targeted task should both be described from the use-case side –if applicable– i.e., which specific problem is being solved (e.g., drug-drug-interaction prediction) as well as from the framing of the problem in the system setting (e.g., link prediction task). Finally, explicitly stating the development maturity of the presented system helps the reader to estimate its state of adaptation as well as its reliability. System Overview Depicting the general processing flow information through the presented system facilitates common understanding of the main processes, without diving into too much detail. Special focus should be laid to distinct and describe the main components in this processing flow, being processing units, i.e., Machine Learning components and Reasoning modules, as well as the data structures (e.g., symbolic data such as KGs, or non-symbolic data such as embeddings) on which the processing units operate. Finally, it is essential to highlight possible diferences in these processing flows in diferent phases of the system, e.g., during training and deployed solution.
System Details To further describe the aforementioned ML components in more detail, the authors should provide information about the model architecture including the base models used (e.g., BERT-base), additional modules (e.g.cross-attention layer), as well as development and training details such as the training procedure (e.g., distantly supervised), loss function and utilized optimizers. For the SW components characteristics such as size and formalism of the SW resource are interesting. Furthermore, the type (e.g., taxonomy / ontology) as well as their semantic exploitation (e.g., only labels / one type of relation are used) provide highly useful information. Finally, any used semantic processors (e.g., reasoners) should be well documented. System Evaluation To increase the reproducibility, evaluation details need to be captured and reported such as pre-processing steps (e.g., hyperparameter tuning), final model parameters , hardware specifications and auditability, such as relevant context information on the system lifecycle, starting from the design phase to the operating system. 3. Methods for SWeMLS Documentation This section describes selected methodologies to facilitate describing and documenting SWeMLS based on the identified relevant information (cf. Section 2).
supervised; self-supervised; semisupervised; unsupervised; reinforcement
labels; hierarchical rels; simple rels; complex structures
low (scripts, prototype); medium (beta, demo); SystemMaturity high (stable, enterprise, tool)
Reasoner; SPARQL query engine
Classical ML: Bayesian; KNN; SVM; Decision Trees; Mixture Models; Regressions;
Markov-process Models; Clusterings; Dim.
Reduction; Formal Concept A.; Rule Learning; Topic Models; SOM; Genetic
Algorithms; Factorization Machines Deep Learning: Transfomers; LSTMs; Trad.
RNN Models; Trad. CNN; GAN; Plain Encoder-based; Trad. FFNN; Matrix
Factorization
Graph Deep Learning: Translational Distance; Rec. GNN; Conv. GNN; Graph
AutoEncoders; Graph FFNN 1..1 1..1 1..1 0..N 1..N
1..N 1..N
Legend:
with cardinality 1..1 1..1 1..1 1..1
subsumption (subClassOf)
General; Admin. & Politics; Geography & Economics; Human Culture & Education; Natural Sciences; News & Social Media; Production of Goods; Software & Tech; Other Text: Analysis; Annotation; QA & conversational; Information Extraction; Information Retrieval; Other Graph: Extension; Creation; Alignment; Other Image and Video; Other Documentation properties: Pattern sub-classes: - Atomic Pattern - T-Pattern - I-Pattern - Fusion Pattern - Y-Pattern - Other SWResourceType: Thesaurus; Taxonomy; Ontology; Dataset; KB; Linked dataset; KG SWResourceSize: <500; 500 -1K; 1K-10K; 10K-100K; 100K-500K; 500K-1M; >1M SWResourceFormalism: owl / owl-2; rdf/rdf-s; other
indirect relations via SHACL constraints
- software - parameter - data - evaluation metrics - data-split - provenance
For describing the overall SWeMLS, we propose the usage of the SWeMLS classification system
introduced in [
Describing SWeMLS Processing Flows
As the processing flow forms one of the most
essential parts in understanding SWeMLS, great efort should be put in its documentation. To
facilitate the description, we propose a visual representation based on existing boxologies, as
they provide an intuitive way of abstraction and documentation. The boxology used in our
conducted SMS is built on the framework introduced in [
The aim of documenting the auditability characteristics
of SWeMLS, is to increase the transparency of design decisions and operational details. For the
ML design phase, Naja et al. [
Future Work. We plan to extend our classification system and to further enhance the machine-readability of the descriptions (e.g., via Open Research Knowledge Graph initiative) and propose an evaluation framework to assess the level of auditability of such systems. Acknowledgments This work is supported by the FFG Project OBARIS (Grant Agreement No 877389).