Large Language Models (LLMs), particularly those based on Generative Pre-trained Transformers (GPT), have achieved strong performance in various natural language tasks. However, LLMs are limited by a knowledge cut-of, so their information is not updated. Common methods for updating LLM knowledge, such as finetuning, retrieval-augmented generation, and machine unlearning, are often resource-intensive and may introduce unintended efects, including the loss of relevant context or conflicts with existing knowledge. Knowledge Editing (KE) ofers a more eficient alternative by enabling precise updates to specific facts without retraining the entire model, while preserving unrelated information. Still, such edits can trigger unexpected ripple efects, known as the Butterfly Efect, where modifying one fact causes errors in related knowledge. In this work, we introduce ButterflyKE, a knowledge graph-based analysis method that probes neighboring knowledge to identify local side efects caused by a single factual update. Using Wikidata as a reference knowledge graph, in ButterflyKE, we extract directly connected triples to provide a structural view of how knowledge propagates after editing. We evaluate three main KE approaches: External Memory-based, Global Optimization-based, and Local Modification-based approaches, using the Llama-3.1-8B-Instruct model. Our findings confirm the presence of the Butterfly Efect in KE, with side efects intensifying as the structural connections increase. To measure this impact, we propose the Butterfly Index, a metric to evaluate editing methods and their influence on surrounding knowledge. ButterflyKE serves as a practical method for extending existing benchmarks and supports a deeper analysis of knowledge integrity in LLM.
In this modern era, Large Language Models (LLMs), particularly those based on Generative Pre-trained
Transformers (GPT), have revolutionized various fields, including Question Answering, Machine
Translation, and Natural Language Inference (NLI). Nevertheless, as black-box models, the complexity of
LLMs presents challenges, as their limited by a knowledge cut-of, so their information is not updated.
In recent years, Knowledge Editing (KE) [
, where one edit disrupts related knowledge. While recent
work focuses on making edits more accurate and precise, limited attention has been given to evaluating
potential side efects. Current KE methods can be broadly categorized into three strategies:
1. External Memory-based Approach: Stores new knowledge externally without changing
internal weights, such as RAG and In-context Knowledge Editing (IKE) [
CEUR
ceur-ws.org
In this work, we introduce ButterflyKE: Butterfly Efects in Knowledge Editing , a method designed to systematically probe the side efects of knowledge editing in LLMs. By leveraging structured knowledge from knowledge graph, it traces how a single factual update may propagate through semantically connected facts. We evaluate these efects using the proposed Butterfly Index , which quantifies the model’s ability to maintain factual correctness in next-hop neighboring knowledge.
To identify and interpret the side efects of Knowledge Editing in Large Language Models, we introduce ButterflyKE , a knowledge graph-based framework that probes next-hop neighboring knowledge to detect localized efects from a single factual update. Instead of constructing a new dataset, we use the public knowledge graph such as Wikidata to enable structural reasoning and trace how edits can propagate through semantically connected facts in the model’s internal knowledge. Figure 1 (a) illustrates the three main components of the framework, while Figure 1(b) illustrates an example of next-hop probing across diferent types of relationships. In this figure, solid black arrows indicate original triples, dashed arrows represent next-hop connections retrieved from the knowledge graph, red arrows denote induced triples inferred from ontological properties, and red arrows marked with a cross indicate removed connections resulting from knowledge edit.
1. Next-Hop Relation Extraction: Given an edit instance in the form of a triple (subject, predicate, object), we first retrieve its adjacent triples from an external knowledge graph using SPARQL queries. Here, adjacency refers to triples that share either the subject entity or the object entity with the edited triple. For example, for the triple [Britney_Spears, :parent, Jamie_P_Spears], adjacent triples include [Jamie_P_Spears, :spouse, Lynn_Spears] and [Britney_Spears, :sibling, Jamie_Lynn_Spears]. This step defines the immediate structural context in which the edit may have side efects. 2. Inherent Relation Induction: After performing the edit by editing the original target entity with a new entity, we expand the neighborhood by enforcing inherent relation constraints derived from ontological properties. Specifically, we consider: inverse relations, if ( 1, , 2) holds, then ( 2, −1, 1) should also hold; symmetric relations, if ( 1, , 2) holds, then ( 2, , 1) should also hold; transitive relations, if ( 1, , 2) and ( 2, , 3) hold, then ( 1, , 3) can be inferred. For example, if Britney is edited to have Errol Musk as a parent, the induced inverse relation makes Britney a child of Errol; if Elon is a sibling of Kimbal, then Kimbal must also be a sibling of Elon; and if Errol is the parent of Elon and Elon is the parent of another entity, then Errol becomes the grandparent of that entity. These induced triples represent the logical consequences of the edit that may introduce inconsistencies or propagate across unrelated domains. 3. Butterfly Efect Measurement
: We aims to probe the impact of a knowledge edit using factual questions derived from next-hop knowledge and inherent relations identified in previous steps. After performing the edit, the model is queried to observe changes in its responses. For example, from [Britney_Spears, :parent, Errol_Musk] we ask “Who is the parent of Britney Spears?”, and from the induced inverse relation [Errol_Musk, :child, Britney_Spears] we ask “Who is the child of Errol Musk?”. By comparing the model’s answers before and after the edit, we identify discrepancies in correctness that signal local disruptions.
To evaluate the side efects of knowledge editing on semantically related information, we introduce the Butterfly Index
(Equation 1). This metric quantifies the degradation in factual accuracy on next-hop knowledge due to the edit by comparing the model’s answers before and after the update.
1 =1 ButterflyIndex = ∑ [( orig( ) = ) − ( edit( ) = )] (1)
Here, orig and edit denote the language model before and after editing, respectively; is the factual question derived from the -th probed triple; is the ground truth answer; and (⋅) is the indicator function, returning 1 if the answer is correct and 0 otherwise. A higher Butterfly Index reflects a greater loss in accuracy on neighboring knowledge, thereby indicating stronger unintended side efects of the edit.
Table 2 presents the performance of representative KE-for-LLMs approaches evaluated on the original CounterFact dataset and its probed counterpart, CounterFact-Probed, which includes next-hop neighboring triples generated using the ButterflyKE framework. All methods achieve high accuracy on CounterFact, confirming their efectiveness at injecting and retrieving the edited knowledge. However, when evaluated on CounterFact-Probed, accuracy drops significantly across all methods, revealing local inconsistencies is introduced by the edit. This degradation is quantified by the Butterfly Index , which measures the diference in accuracy before and after editing on next-hop knowledge. For example, IKE achieves a perfect editing accuracy of 1.0, but its accuracy on neighboring facts drops to 0.511, resulting in a Butterfly Index of 0.489. Similarly, ROME drops from 0.87 to 0.26, yielding the highest Butterfly Index of 0.61 among all methods. These results indicate that although edits are successful in isolation, they often disrupt related factual knowledge embedded within the model.
These findings demonstrate the presence of the Butterfly Efect in knowledge editing. A localized factual change can unintentionally afect semantically related information within the model. This observation reveals a key limitation of current knowledge editing techniques, which often fail to maintain the broader contextual consistency of the model’s internal knowledge. The Butterfly Index helps bridge this gap by ofering a principled metric that captures not only the factual accuracy but also the semantic stability of the model after a edit.
In this study, we presented ButterflyKE , a framework to evaluate local side efects of KE-for-LLMs. By enriching the CounterFact dataset with next-hop neighboring triples, we constructed CounterFactProbed, enabling probing of unintended impacts on semantically related knowledge. To quantify these efects, we proposed the Butterfly Index , measuring accuracy diferences on surrounding facts before and after editing. Experimental results show that while KE methods succeed in updating the target information, they vary significantly in their ability to preserve adjacent facts. In particular, they experience substantial drops in accuracy on neighboring triples, revealing local disruptions despite successful edits. These findings confirm the presence of the Butterfly Efect in KE. This highlights a key limitation of current approaches and emphasizes the need for methods that ensure both factual precision and semantic stability. In future work, we will broaden the evaluation to diverse editing techniques and domains, and extend analysis to advanced foundation models such as ChatGPT, DeepSeek, and Gemini. We also plan to investigate deeper graph structures and multi-hop interactions to better understand interference mechanisms and guide the design of more robust editing strategies.
During the preparation of this work, the authors used ChatGPT-4o to: Grammar, paraphrase, and reword. After using this tool, the authors reviewed and edited the content as needed and assumed full responsibility for the publication’s content.