Knowledge editing has emerged as a promising way to update knowledge in large-scale language models (LLMs) eficiently. However, current knowledge editing methods focus on undiferentiated factual knowledge, neglecting the significance of hierarchical structured knowledge editing. Moreover, cognitive science has revealed the importance of hierarchical knowledge for human learning. This poster introduces hierarchical knowledge editing for instance-to-concept relationship. Through Layered Distillation strategy, we perform knowledge distillation between the original and edited models, thereby preserving instance-to-concept relationship hierarchical knowledge in the original model. Experimental results demonstrate that integrating our strategy with existing knowledge editing methods enhances the performance of hierarchical knowledge editing.
CEUR ceur-ws.org
With the performance improvement and wide application of large language models, the
problems of LLMs, such as providing outdated, erroneous, or toxic information, have become the
focus of criticism. Retraining LLMs to address these issues takes time and exertion. In contrast,
knowledge editing ofers a low-cost way to update trained models. This has made the
development of eficient and reliable knowledge editing methods for LLMs a key area of research[
LLMs memorize various hierarchical knowledge, hierarchical knowledge editing task
includes editing both instance-to-concept and other inter-conceptual relationships hierarchical
knowledge. Focusing on instance-to-concept relationship, we want to retain the integrity of this
relationship in the original model after editing concept. Consider a simple instance-to-concept
relationship hierarchical knowledge in LLMs: instance “tiger” belongs to concept “feline,” which
is defined as a carnivorous mammal known for flexible body and sharp claws. In the case where
the definition of “feline” is edited to winged animal, since “tiger” is wingless, human cognition
will naturally assume that “tiger” no longer belongs to “feline,” which means that we do not want
this editing to modify the hierarchical knowledge of “tiger” belongs to “feline” in the original
LLMs[
In this poster, we define hierarchical Knowledge Editing task with corresponding metric, design Layered Distillation strategy for preserving instance-to-concept relationship hierarchical knowledge, and present experimental evidence of its meaningful efectiveness in enhancing hierarchical knowledge editing.
LLMs memorize various hierarchical knowledge, Hierarchical Knowledge Editing task includes managing both instance-to-concept and other inter-conceptual relationships. Focusing on instance-to-concept relationship, our goal is to design an editing approach that retains the integrity of the relationship when concept definitions are updated.
Hierarchical knowledge editing task is formally defined as: given a concept = (, ) , where is the concept name and is the concept definition, and a set of instances , all instances in belong to concept , denoted ∈ . When the definition is edited to ∗, resulting in the modified concept ∗ = (, ∗), a great hierarchical knowledge editing approach must ensure that the modification of concept does not result in ∈ ∗, maintain the integrity of , minimize instance migration, and avoid unacceptable changes to the original model.
We introduce knowledge distillation to ensure the edited model inherits the original model’s
hierarchical knowledge[
To more efectively assess the impact of our editing approach on hierarchical knowledge, we devise Instance Retention(IR) metric. Its definition is as follows:
1 ∈ = 1 − ∑[(, ∗) + | (, ) − (, ∗)|] (1)
Instance Retention measures the proportion of instance-to-concept relationship knowledge
preserved in the edited model. It is determined by two functions: (, ), which indicates
whether instance belongs to concept ( = 1 if it does, = 0 if it does not). (, ), used
when instance cannot determine whether it belongs to concept ( = 1 in this case, = 0
otherwise). These functions are applied using the reasoning capability of the LLM[
Main results of the Hierarchical Knowledge Editing experiment. “+LD” stands for combined with our strategy. Bold results denote optimal performance.
Utilizing the ConceptEdit[
Table 1 results indicate that both editing methods struggle with hierarchical knowledge editing in both models, but the IR of ROME method is significantly higher. Meanwhile, Layered Distillation strategy with these methods enhances IR without notably altering other metrics, particularly for ROME editing TinyLlama. This enhancement validates the eficacy of our strategy for editing instance-to-concept relationship hierarchical knowledge. Furthermore, the relatively modest enhancement observed in the MEMIT+LD configuration can be attributed to the editing across multiple MLP layers, which is responsible for the suboptimal performance of the GPT2-XL model, but concurrently ofers a higher degree of Locality.
This poster proposes Hierarchical Knowledge Editing for LLMs and Layered Distillation strategy to enhance existing knowledge editing methods for editing instance-to-concept relationship hierarchical knowledge. Our experiments initially demonstrate the eficacy of Layered Distillation. Future research will broaden validation to additional methods and larger LLMs. Editing other hierarchical relationship knowledge will also be further explored.