Knowledge editing has emerged as a promising way to update knowledge in large-scale language models (LLMs) efficiently. However, current knowledge editing methods focus on undifferentiated factual knowledge, neglecting the significance of hierarchical structured knowledge editing. Moreover, cognitive science has revealed the importance of hierarchical knowledge for human learning. This poster introduces hierarchical knowledge editing for instance-to-concept relationship. Through Layered Distillation strategy, we perform knowledge distillation between the original and edited models, thereby preserving instance-to-concept relationship hierarchical knowledge in the original model. Experimental results demonstrate that integrating our strategy with existing knowledge editing methods enhances the performance of hierarchical knowledge editing.
With the performance improvement and wide application of large language models, the problems of LLMs, such as providing outdated, erroneous, or toxic information, have become the focus of criticism. Retraining LLMs to address these issues takes time and exertion. In contrast, knowledge editing offers a low-cost way to update trained models. This has made the development of efficient and reliable knowledge editing methods for LLMs a key area of research [1]. However, most existing LLM knowledge editing methods do not differentiate between types of knowledge and primarily concentrate on editing factual knowledge individually, which is inefficient for LLMs with a massive number of parameters that store vast amounts of knowledge. Moreover, there is a lack of effective retention of structured hierarchical information within LLMs. Based on this, this poster proposes hierarchical knowledge editing.
LLMs memorize various hierarchical knowledge, hierarchical knowledge editing task includes editing both instance-to-concept and other inter-conceptual relationships hierarchical knowledge. Focusing on instance-to-concept relationship, we want to retain the integrity of this relationship in the original model after editing concept. Consider a simple instance-to-concept relationship hierarchical knowledge in LLMs: instance "tiger" belongs to concept "feline, " which is defined as a carnivorous mammal known for flexible body and sharp claws. In the case where the definition of "feline" is edited to winged animal, since "tiger" is wingless, human cognition will naturally assume that "tiger" no longer belongs to "feline, " which means that we do not want this editing to modify the hierarchical knowledge of "tiger" belongs to "feline" in the original LLMs [2]. Thus, it is vital to maintain the original instance-to-concept relationship hierarchical knowledge when editing LLMs. While a recent work proposes editing conceptual knowledge, which focuses on the effects of modifying concept definitions within LLMs [2], hierarchical knowledge editing concentrates on evolving editing approaches that preserve hierarchical knowledge within LLMs.
In this poster, we define hierarchical Knowledge Editing task with corresponding metric, design Layered Distillation strategy for preserving instance-to-concept relationship hierarchical knowledge, and present experimental evidence of its meaningful effectiveness in enhancing hierarchical knowledge editing.
LLMs memorize various hierarchical knowledge, Hierarchical Knowledge Editing task includes managing both instance-to-concept and other inter-conceptual relationships. Focusing on instance-to-concept relationship, our goal is to design an editing approach that retains the integrity of the relationship when concept definitions are updated.
Hierarchical knowledge editing task is formally defined as: given a concept 𝐶 = (𝑐, 𝑑), where 𝑐 is the concept name and 𝑑 is the concept definition, and a set of instances 𝐼, all instances 𝑖 in 𝐼 belong to concept 𝐶, denoted 𝑖 ∈ 𝐶. When the definition 𝑑 is edited to 𝑑 * , resulting in the modified concept 𝐶 * = (𝑐, 𝑑 * ), a great hierarchical knowledge editing approach must ensure that the modification of concept does not result in 𝑖 ∈ 𝐶 * , maintain the integrity of 𝐼, minimize instance migration, and avoid unacceptable changes to the original model.
We introduce knowledge distillation to ensure the edited model inherits the original model's hierarchical knowledge [3]. We first use Location-Then-Edit as the base editing method [1,4,5]. This method targets specific model layers for editing, enabling focused distillation of the original and edited model layers, rather than the entire model. Layered Distillation is updating the model layers again using these original and edited model layers as input, with Mean Square Error as the distillation loss. The specific process is shown in Figure 1. The base method edits the concept definition in the original model, resulting in instance migration. Updating the edited model again through the Layered Distillation strategy retains the instance-to-concept relationship in the original model.
To more effectively assess the impact of our editing approach on hierarchical knowledge, we devise Instance Retention(IR) metric. Its definition is as follows:
Instance Retention measures the proportion of instance-to-concept relationship knowledge preserved in the edited model. It is determined by two functions: 𝐺(𝑖, 𝐶), which indicates whether instance 𝑖 belongs to concept 𝐶 (𝐺 = 1 if it does, 𝐺 = 0 if it does not). 𝐻 (𝑖, 𝐶), used when instance 𝑖 cannot determine whether it belongs to concept 𝐶(𝐻 = 1 in this case, 𝐻 = 0 otherwise). These functions are applied using the reasoning capability of the LLM [2]. We also use Reliability(Rel.), Generalization(Gen.), and Locality(Loc.) as comprehensive metrics to evaluate the success, scope, and impact of knowledge editing [1]. Utilizing the ConceptEdit [2], derived from the DBpedia ontology dataset, we conducted experiments on the open-source LLMs GPT2-XL (1.5B) [6]and TinyLlama (1.1B) [7] using the ROME [4] and MEMIT [5] methods combined with Layered Distillation(LD) on GeForce RTX 4090. ConceptEdit's Intra and Inter modules represent modifications to concepts within and between superclasses.
Table 1 results indicate that both editing methods struggle with hierarchical knowledge editing in both models, but the IR of ROME method is significantly higher. Meanwhile, Layered Distillation strategy with these methods enhances IR without notably altering other metrics, particularly for ROME editing TinyLlama. This enhancement validates the efficacy of our strategy for editing instance-to-concept relationship hierarchical knowledge. Furthermore, the relatively modest enhancement observed in the MEMIT+LD configuration can be attributed to the editing across multiple MLP layers, which is responsible for the suboptimal performance of the GPT2-XL model, but concurrently offers a higher degree of Locality.
This poster proposes Hierarchical Knowledge Editing for LLMs and Layered Distillation strategy to enhance existing knowledge editing methods for editing instance-to-concept relationship hierarchical knowledge. Our experiments initially demonstrate the efficacy of Layered Distillation. Future research will broaden validation to additional methods and larger LLMs. Editing other hierarchical relationship knowledge will also be further explored.