Introduction

Nowadays, with the widespread of misinformation, there is a growing demand for Fact Verification to determine the veracity of claims. In conventional Fact Verification, evidence is retrieved from external sources (Retriever), and the judgment is predicted by LLM (Reader) based on the evidence [1]. However, when LLM is used as the Reader, the inference process that leads to the final judgment becomes a black box, lacking explainability. On the other hand, a method has been proposed to perform rule-based judgments incorporating logical symbols as a Reader, rather than using LLM for judgment [2]. This method has high accuracy on FEVER [3], a representative dataset of Fact Verification, while enhancing explainability. In this paper, we propose a rule-based approach similar to [2], but utilizes a knowledge graph. We created rules to deal with the five reasoning types of Fact Verification defined in FACTKG [4]. A key difference from [2] is that external sources are interpretable knowledge graphs, not text. Experimental results with FACTKG show that rule-based judgments achieve a sufficient level of accuracy.

FACTKG

FACTKG (Fact Verification via Reasoning on Knowledge Graphs) [4] is a dataset for Fact Verification that leverages knowledge graph. It classifies claims into four reasoning types based on the structure of the knowledge graph (Table 1). For each type, Negation is also created by adding negations (e.g. 'not' or 'never'), resulting in a total of five types.

• One-Hop: Claims can be verified with a single knowledge triple. In Table 1, if triple (AIDAStella, Builder, Meyer Werft) exists, it is labeled as SUPPORTED.

IJCKG 2023 Poster and Demo track Envelope 235x075x@gsuite.kobe-u.ac.jp (Y. Momii); takigu@kobe-u.ac.jp (T. Takiguchi); ariki@kobe-u.ac.jp (Y. Ariki)

• Conjunction: Claims can be verified with multiple knowledge triples. Both (AIDAstella, Operator, AIDA Cruises) and (AIDAstella, Builder, Meyer Werft) need to exist for it to be labeled as SUPPORTED. • Existence: Claims can be determined based on whether a specific relation exists. If Meyer Werft has the parentCompany, it is labeled as SUPPORTED. • Multi-Hop: Entities in Conjunction are transformed into common nouns. If both (AIDAstella, Builder, x) and (x, location, Papenburg) exist, it is labeled as SUPPORTED.

Table 1

Four different reasoning types of FACTKG cited from [4].r1: parentCompany, r2: Builder, r3: Operator, r4: location, m: Meyer Werft, s: AIDAstella, c: AIDA Cruise, p: Papenburg.

Reasoning type Claim Example Graph

One-Hop AIDAstella was built by Meyer Werft.

Conjunction AIDA Cruise line operated the AIDAstella which was built by Meyer Werft.

Existence Meyer Werft had a parent company.

Multi-Hop

AIDAstella was built by a company in Papenburg.

All claims are labeled as SUPPORTED or REFUTED. REFUTED claims are created by replacing an entity in SUPPORTED claims with another entity. In the case of One-Hop, relation substitution can also be done. Negation creation depends on the type. For One-Hop and Existence, they are created by adding negations into the claim and inverting the label. In the case of Conjunction, it is labeled as SUPPORTED only if negations are added to all parts with substituted entities. In Multi-Hop, the label is based on the actual knowledge graph. For example, "AIDAstella was built by a company, not in Papenburg" is labeled as SUPPORTED if (AIDAstella, Builder, x) and (x, location, y) exist, and y is not Papenburg. In Negation with Conjunction (or Multi-Hop), the number of entities are fixed at 3 (or 2), and the number of hops is 2. The rules are applied under those conditions. However, extensions are possible with sentence parsing.

Proposed Method

Our proposed method consists of three predictors(Relation, Reasoning type, Negation) and manually created rules. Appropriate rules are selected based on predictors' result. All predictors employ BERT(bert-base-uncased) [5]. Relation Predictor takes"Claim[SEP]Entity" as input and predicts relations as multi-labels for all entities in the claim. Then they are combined to form Relation Set 𝑅. Reasoning type Predictor classifies the claim based on the categories in Table 1. Negation Predictor predicts whether negations are added.

Rule-based judgment

• One-Hop (Figure 1(a)): For this type, we verify whether the entities in the claim are directly connected each other. If not, we determine that the claim is based on incorrect knowledge triples and predict it as REFUTED. Otherwise, we verify if they're connected by the relations included in 𝑅, and if so, we predict it as SUPPORTED. This decision allows for handling cases where the claim incorrectly represents entities relationship. for subgraph by exploring from one entity in the claim along candidate paths, and if all entities appear in subgraph, we predict it as SUPPORTED. Note that the hop count of subgraph is fixed at 3. This is because the maximum hop count in the dataset is 3, and the final judgment is not based on LLM. We need not to consider the size of subgraph.

Rule-based judgment with negated claims

• One-Hop, Existence: We reverse the predicted labels without negation.

• Conjunction: First, we examine the #number of entities that don't connect with other entities. #0: We predict it as REFUTED. #1: In the case of third line of Table 1, either of the entities at both ends is replaced. #3:This occurs when the central entity is replaced. In cases #1 and #3, subsequently, the negations and the position of the entity are identified through keyword matching. We predict it as SUPPORTED only when negations are present in all positions related to the substituted entity. • Multi-Hop: In this case, we search for edge entities with predicted relations of each entity. Then common entities are excluded. The position of the entities and negations are determined in the claim, and 𝑁 𝑒 , entities not related to replaced entity is identified. If 𝑁 𝑒 has one or more excluded edge entities, we predict it as SUPPORTED.

Experiment

The baseline (Model-Based) refers to FACTKG's baseline. Relation prediction is performed in a manner similar to the proposed method, and the number of hops required for graph traversal is also predicted. Using the acquired subgraph, the final judgment is predicted by classifier(BERT(bert-base-cased) with linear layer). FACTKG is used to train our predictors and baseline's classifier as the dataset, but the test data lacks the necessary subgraph for judgment, so we divided the validation data into two parts to create new validation and test data. Besides the simple accuracy (LA), we report the evaluation of Evidence Enhanced Label Accuracy (EELA), which considers relation prediction errors as judgment errors. Furthermore, Oracle Label Accuracy (OLA), which indicates the accuracy when correct relations are provided. Table 2 shows the results. The proposed method performed better in many of the reasoning types.

Conclusion

We proposed a rule-based method for fact verification using a knowledge graph, aiming for better explainability and accuracy. Difference between LA and EELA of model-based method reveals that it may predict correct results with incorrect reasoning. Future work will focus on quantitative assessments of explainability. Furthermore, we will incorporate grammatical interpretation to avoid errors in determining relations related to negations.