In this paper, we propose a method for performing Fact Verification using the structure of a knowledge graph and pre-defined rules. We focus on the classification of claims into five reasoning types based on the knowledge graph in FACTKG and manually create classification-specific rules. This method allows us to create rules that are independent of entities and relations. Experimental results with FACTKG show that rule-based judgments achieve a suficient level of label accuracy.
CEUR ceur-ws.org
FACTKG (Fact Verification via Reasoning on Knowledge Graphs)[
• One-Hop: Claims can be verified with a single knowledge triple. In Table 1, if triple (AIDAStella, Builder, Meyer Werft ) exists, it is labeled as SUPPORTED. nEvelop-O IJCKG 2023 Poster and Demo track © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings • Conjunction: Claims can be verified with multiple knowledge triples. Both ( AIDAstella, Operator, AIDA Cruises) and (AIDAstella, Builder, Meyer Werft ) need to exist for it to be labeled as SUPPORTED. • Existence: Claims can be determined based on whether a specific relation exists. If Meyer
Werft has the parentCompany, it is labeled as SUPPORTED. • Multi-Hop: Entities in Conjunction are transformed into common nouns. If both (AIDAstella,
Builder, x) and (x, location, Papenburg) exist, it is labeled as SUPPORTED.
All claims are labeled as SUPPORTED or REFUTED. REFUTED claims are created by replacing an entity in SUPPORTED claims with another entity. In the case of One-Hop, relation substitution can also be done. Negation creation depends on the type. For One-Hop and Existence, they are created by adding negations into the claim and inverting the label. In the case of Conjunction, it is labeled as SUPPORTED only if negations are added to all parts with substituted entities. In Multi-Hop, the label is based on the actual knowledge graph. For example,“AIDAstella was built by a company, not in Papenburg” is labeled as SUPPORTED if (AIDAstella, Builder, x) and (x, location, y) exist, and y is not Papenburg. In Negation with Conjunction (or Multi-Hop), the number of entities are fixed at 3 (or 2), and the number of hops is 2. The rules are applied under those conditions. However, extensions are possible with sentence parsing.
Our proposed method consists of three predictors(Relation, Reasoning type, Negation) and
manually created rules. Appropriate rules are selected based on predictors’ result.
All predictors employ BERT(bert-base-uncased)[
Rule-based judgment • One-Hop (Figure 1(a)): For this type, we verify whether the entities in the claim are directly connected each other. If not, we determine that the claim is based on incorrect knowledge triples and predict it as REFUTED. Otherwise, we verify if they’re connected by the relations included in , and if so, we predict it as SUPPORTED. This decision allows for handling cases where the claim incorrectly represents entities relationship. • Conjunction (Figure 1(a)): Like One-Hop, we check if all entities are connected or not. • Existence (Figure 1(b)): In this type, we retrieve all relations possessed by the entity in the claim, and if there is any common relation with , we predict it as SUPPORTED. • Multi-Hop (Figure 1(c)): We arrange to generate candidate paths. Then, we search for subgraph by exploring from one entity in the claim along candidate paths, and if all entities appear in subgraph, we predict it as SUPPORTED. Note that the hop count of subgraph is fixed at 3. This is because the maximum hop count in the dataset is 3, and the final judgment is not based on LLM. We need not to consider the size of subgraph. Rule-based judgment with negated claims • One-Hop, Existence: We reverse the predicted labels without negation. • Conjunction: First, we examine the #number of entities that don’t connect with other entities. #0: We predict it as REFUTED. #1: In the case of third line of Table 1, either of the entities at both ends is replaced. #3:This occurs when the central entity is replaced. In cases #1 and #3, subsequently, the negations and the position of the entity are identified through keyword matching. We predict it as SUPPORTED only when negations are present in all positions related to the substituted entity. • Multi-Hop: In this case, we search for edge entities with predicted relations of each entity. Then common entities are excluded. The position of the entities and negations are determined in the claim, and , entities not related to replaced entity is identified. If has one or more excluded edge entities, we predict it as SUPPORTED.
The baseline (Model-Based) refers to FACTKG’s baseline. Relation prediction is performed in a manner similar to the proposed method, and the number of hops required for graph traversal is also predicted. Using the acquired subgraph, the final judgment is predicted by classifier(BERT(bert-base-cased) with linear layer). FACTKG is used to train our predictors and baseline’s classifier as the dataset, but the test data lacks the necessary subgraph for judgment, so we divided the validation data into two parts to create new validation and test data. Besides the simple accuracy (LA), we report the evaluation of Evidence Enhanced Label Accuracy (EELA), which considers relation prediction errors as judgment errors. Furthermore, Oracle Label Accuracy (OLA), which indicates the accuracy when correct relations are provided. Table 2 shows the results. The proposed method performed better in many of the reasoning types.
We proposed a rule-based method for fact verification using a knowledge graph, aiming for better explainability and accuracy. Diference between LA and EELA of model-based method reveals that it may predict correct results with incorrect reasoning. Future work will focus on quantitative assessments of explainability. Furthermore, we will incorporate grammatical interpretation to avoid errors in determining relations related to negations. Acknowledgements This work was supported in part by JSPS KAKENHI (Grant No. JP21H00906).