Knowledge graphs (KGs) require continual updates as new information emerges, but neural embedding models sufer from catastrophic forgetting when learning new tasks sequentially. We evaluate Elastic Weight Consolidation (EWC), a regularization-based continual learning method, on KG link prediction using TransE embeddings on FB15k-237. Across multiple experiments with five random seeds, we find that EWC reduces catastrophic forgetting from 12.62% to 6.85%, a 45.7% reduction compared to naive sequential training. We observe that the task partitioning strategy afects the magnitude of forgetting: relation-based partitioning (grouping triples by relation type) exhibits 9.8 percentage points higher forgetting than randomly partitioned tasks (12.62% vs 2.81%), suggesting that task construction influences evaluation outcomes. While focused on a single embedding model and dataset, our results demonstrate that EWC efectively mitigates catastrophic forgetting in KG continual learning and highlight the importance of evaluation protocol design.
Knowledge graphs (KGs) depict structured information as networks of entities and their relations [
Catastrophic forgetting occurs when neural networks, trained sequentially on multiple tasks,
experience significant performance degradation on earlier tasks after learning new ones [
We investigate how Elastic Weight Consolidation (EWC) [
Our experiments reveal that EWC substantially reduces catastrophic forgetting. On relation-based partitioned tasks, naive sequential training results in 12.62% forgetting (measured as MRR degradation from post-task performance), while EWC with regularization strength = 10 reduces this to 6.85%, a 45.7% reduction. This demonstrates that regularization-based continual learning efectively preserves KG embeddings across sequential tasks.
We also observe that the task partitioning strategy significantly afects the measured forgetting. Naive sequential training on relation-based tasks exhibits 12.62% forgetting, compared to only 2.81% on randomly partitioned tasks, a 9.8%point diference. This suggests that evaluation protocols, particularly how tasks are constructed from datasets, influence continual learning metrics and should be carefully explored when it comes time to design the study.
Our study focuses on TransE embeddings for the FB15k-237 dataset across four relation-based tasks. While this scope limits generalizability, it provides rigorous evidence that EWC reduces catastrophic forgetting in KG continual learning and raises essential questions about task construction in continual learning evaluation.
This work addresses a critical challenge for knowledge graph-based AI agents: maintaining knowledge representations as new information arrives. As agents increasingly rely on KG-based memory and reasoning, the ability to incorporate new knowledge while preserving existing facts becomes essential for long-term autonomous operation.
KG Embeddings. TransE [
Continual Learning. Methods for mitigating catastrophic forgetting include regularization
approaches like EWC [
KG Continual Learning. Prior work has explored continual learning for KGs across various
contexts. Wang et al. [
A KG is = (ℰ , ℛ, ) where ℰ is the set of entities, ℛ is the set of relations, and ⊆ ℰ × ℛ × ℰ the set of true triples. TransE learns embeddings h, r, t ∈ R by minimizing: ℒ = ∑︁
∑︁ (ℎ,,)∈ (ℎ′,,′)∈ ′ max(0, + (h + r, t) − (h ′ + r, t′)) where ′ contains negative samples, (·, ·) is L2 distance, and is the margin.
In continual learning, we partition into tasks 1, . . . , and train sequentially. After training on task , we measure performance on task ≤ . Forgetting for task after learning task is: = − for > (3) (4) (5)
EWC protects important parameters by adding a regularization term to the loss when training on task : where *,1 are optimal parameters after task −1 , is the Fisher Information diagonal approximation, and contr− ols regularization strength. The Fisher Information Matrix diagonal is:
ℒEWC = ℒ + 2 ∑︁ ( − *,−1 )2 = E(ℎ,,)∼ −1 [︃︂( log (|; ) )︂ 2]︃ We report average forgetting at the end of training:
We compute this empirically using all triples from the previous task, processed in mini-batches (see Appendix A for implementation details).
We evaluate two task partitioning strategies to assess how task construction afects catastrophic forgetting:
Relation-based partitioning: We partition FB15k-237 by grouping all triples that share the same relations. To create balanced task sizes, we sort the 237 relations by frequency (number of triples per relation) and assign them to four tasks in round-robin order: the most frequent relation goes to Task 1, the second-most to Task 2, the third-most to Task 3, the fourth-most to Task 4, the fifth-most back to Task 1, and so forth. This ensures each task receives approximately 59 relations with a mixture of common and rare relation types, while maintaining relation-level coherence—all triples with the same relation appear in the same task.
Random partitioning: We randomly shufle all 272,115 training triples and divide them into four equal chunks of approximately 68,000 triples each. This distributes relation types across all tasks, creating relation-level overlap—most relations appear in multiple tasks. The key diference: relationbased partitioning creates distinct distribution shifts between tasks (each task focuses on diferent relation types). In contrast, random partitioning minimizes distribution shift (each task is a representative sample of all relations). This allows us to isolate the efect of task boundary definition on the dificulty of continual learning.
Dataset and Partitioning. We use FB15k-237 [
Model Configuration. We use TransE with 50-dimensional embeddings, margin = 1.0 , and
L2 distance. Training uses the Adam optimizer [
Methods Evaluated. We compare naive sequential training (no continual learning) with EWC at multiple regularization strengths ( ∈ {0.1, 1.0, 10.0} ), EWC combined with experience replay (500 examples per task), and replay-only baselines (random and wave-based sampling).
Evaluation Protocol. We use Mean Reciprocal Rank (MRR) for link prediction, computing filtered rankings that exclude known actual triples. For each task , we record MRR immediately after training ( ) and after each subsequent task ( for > ). We run five random seeds (42, 123, 456, 789, 2024) and report means and standard deviations.
Hardware. Experiments ran on NVIDIA RTX 3070 Ti (8GB) with 20 hours total computation for 80 experiments (8 methods × 5 seeds × 2 partitioning strategies).
EWC with = 10 achieves the best performance, reducing forgetting to 6.85% (std 0.33%), a 45.7% reduction compared to naive training. This demonstrates that regularization-based protection of important parameters efectively mitigates catastrophic forgetting in KG continual learning. Final MRR also improves from 0.206 to 0.242, indicating that EWC preserves not only previous task performance but also maintains overall embedding quality.
Interestingly, replay-based methods underperform. Random replay achieves 13.78% forgetting, worse than naive training, suggesting that simply revisiting old examples without principled parameter protection may interfere with learning. Wave-based replay performs similarly (12.54% forgetting). Combining EWC with wave replay (9.91% forgetting) improves over replay alone but underperforms pure EWC, indicating that regularization is the primary driver of performance.
Figure 1 visualizes these results, showing a clear separation between EWC and other methods.
We compare relation-based and random partitioning strategies to assess whether task construction afects measured forgetting. Table 2 and Figure 2 show results.
Relation-based Random
Naive Forgetting (%) EWC Forgetting (%) 12.62 ± 0.35 2.81 ± 0.34 9.81 pp 6.85 ± 0.33 5.08 ± 0.22 1.77 pp
Relation-based partitioning results in substantially higher forgetting during naive training (12.62% vs 2.81%), a 9.8 percentage-point diference. We hypothesize this occurs because relation-based partitioning creates task coherence: each task focuses on a distinct subset of relation types (approximately 59 relations
Catastrophic Forgetting on Relation-Based Task Split 12.6% 10.4% 7.5% 6.8% 13.8% per task), inducing larger distribution shifts when transitioning between tasks. When the model moves from Task 1’s relations to Task 2’s completely diferent relation set, the parameter updates required are more disruptive to previously learned representations. In contrast, random partitioning creates relationlevel overlap: each task contains a representative sample of most relation types, so the distribution shift between tasks is minimal. This naturally regularizes learning, as the model continually encounters all relation types across tasks. This observation suggests that task construction significantly influences the dificulty of continual learning and that evaluation protocols should explicitly consider and report partitioning strategies.
Notably, EWC reduces this gap: the diference between relation-based and random partitioning under EWC is only 1.77 percentage points (6.85% vs 5.08%), suggesting that efective continual learning methods can generalize across diferent task construction approaches.
KG embeddings have structured parameter spaces where specific dimensions encode semantic properties. The Fisher Information Matrix identifies parameters critical for encoding relation types and entity characteristics learned in previous tasks. By protecting these parameters, EWC enables new-task learning while preserving the semantic structure of the embedding space.
The superior performance of EWC compared to replay methods suggests that principled parameter protection is more efective than simply revisiting old examples when working memory (replay bufer) is limited. This aligns with neuroscience findings that synaptic consolidation, rather than replay alone, enables long-term memory retention.
Our EWC forgetting rate (6.85%) on relation-based partitioned tasks demonstrates efective continual
learning for KG link prediction. While direct comparison with prior work is challenging due to
diferences in datasets and evaluation protocols, our results are consistent with EWC’s performance on
image classification tasks [
The task partitioning efect we observe (a 9.8 percentage-point diference) highlights an important consideration for continual learning evaluation: reported forgetting rates depend on how tasks are constructed. This suggests that future work should explicitly consider task construction methodology when designing experiments and reporting results.
Our study has several limitations that define the scope and future directions.
We evaluate only TransE on FB15k-237 across four tasks, a deliberate choice given consumer-grade GPU constraints (NVIDIA RTX 3070 Ti, 8GB) that enabled rigorous multi-seed experimentation within feasible time scales. Results may not generalize to: (1) complex embedding methods (RotatE, ComplEx, TuckER); (2) other datasets (WN18RR, YAGO, Wikidata); (3) more extended task sequences (10+ tasks). We compare EWC with basic replay baselines, but do not exhaustively benchmark all continual learning methods. Our relation-based partitioning uses round-robin frequency assignment; alternative strategies (entity-based, domain-based) warrant investigation.
Experiments were constrained to academic resources without HPC clusters or large-scale GPU infrastructure. While this motivated our focused design, a comprehensive multi-model evaluation would require substantially more resources. We view this as proof-of-concept evidence for future large-scale studies.
Future work should evaluate EWC across multiple embedding methods and datasets to assess generalizability. Scaling studies with 10+ tasks would reveal the long-term dynamics of continual learning. Systematic investigation of task construction strategies could formalize the relationship between task partitioning and forgetting. Additionally, combining EWC with more sophisticated replay strategies or architectural approaches may yield further improvements.
A particularly promising direction is to evaluate continual learning methods on educational KGs,
where knowledge evolves as curriculum content updates and student learning data accumulates. Recent
work has highlighted the potential of open-source LLMs to transform educational contexts through
adaptive and personalized learning [
Neuromorphic approaches using spiking neural networks with spike-timing-dependent plasticity
ofer promising directions for extending this work. Building on prior work demonstrating SNN-based
relational inference and knowledge representation for knowledge graphs [
While our preliminary experiments on consumer-grade GPUs were inconclusive, a comprehensive evaluation using specialized neuromorphic hardware (Intel Loihi, IBM TrueNorth, SpiNNaker) could reveal whether biological learning mechanisms provide advantages for continual learning in knowledge graphs. The structured, relational nature of KGs may align well with neuromorphic computation, particularly for consolidating knowledge across tasks. Given EWC’s success through parameterwise importance weighting, bio-inspired plasticity mechanisms that selectively strengthen or weaken synaptic connections based on usage patterns warrant thorough exploration with appropriate hardware infrastructure.
We evaluated EWC for KG continual learning using TransE embeddings on FB15k-237. Across multiple experiments with five random seeds, we found that EWC reduces catastrophic forgetting from 12.62% to 6.85%, a 45.7% reduction compared to naive sequential training. This demonstrates that regularizationbased continual learning efectively preserves KG embeddings across sequential tasks.
We also observed that the task partitioning strategy significantly afects measured forgetting: relationbased partitioning exhibits 9.8 percentage points higher forgetting than randomly partitioned tasks for naive training. This suggests that the evaluation protocol’s design, particularly the task-construction methodology, influences continual learning measurements and should be carefully considered in experimental design.
While our study focuses on a single embedding model and dataset, it provides rigorous evidence for EWC’s efectiveness and raises essential questions about evaluation methodology in KG continual learning.
We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC), Alberta Innovates, Alberta Advanced Education, and Athabasca University, Canada. We would also like to thank the reviewers for their suggestions on how to improve this work.
During the preparation of this work, the author(s) used Grammarly and Claude (Anthropic) for Grammar and spelling checks.
We compute the Fisher Information Matrix diagonal after training each task using all triples from that task, processed in mini-batches of 256. For each batch , we compute the empirical Fisher approximation: squared gradients across all training examples from the previous task.
where is the total number of batches in the task and ℒ is the loss on batch . This accumulates During training on task , we apply the EWC penalty for all previous tasks: ≈ =1
vs 2.81% naive). We hypothesize that random partitioning naturally distributes relation types across tasks, reducing interference. Strong regularization may over-constrain parameters, preventing necessary adaptation. This suggests that optimal regularization strength depends on task construction.
EWC performance across tested regularization strengths.
For relation-based partitioning, = 10 achieves the lowest forgetting. For random partitioning, weaker regularization ( = 0.1 ) performs best. This suggests that optimal regularization strength depends on task construction: relation-grouped tasks require stronger protection of essential parameters, while randomly distributed tasks benefit from more flexibility.
EWC with = 10 occupies the optimal region (low forgetting, competitive performance). Replay methods cluster in the high-forgetting, low-performance region. This visualization confirms that EWC achieves superior trade-ofs compared to alternative approaches.
Code Structure. Experiments implemented in PyTorch 1.13.
Negative Sampling. We use uniform random tail corruption with a 1:1 ratio of positive to negative samples. For each positive triple (ℎ, , ), we generate one negative sample (ℎ, , ′) by randomly replacing the tail entity ′ sampled uniformly from ℰ . We use unfiltered negative sampling during training (negative samples may coincidentally be true triples), but employ filtered evaluation for link prediction metrics.
Hardware. Experiments ran on NVIDIA RTX 3070 Ti (8GB) with approximately 18-20 hours total computation for all experimental runs (8 methods × 5 seeds × 2 partitioning strategies).
Task Retention Matrix: EWC (λ=10)
Task 2 Task 3
Evaluation Task Task 4 0.40 0.00 0.20 RR
M
The heatmap shows that EWC maintains relatively stable performance across tasks, with limited degradation on earlier tasks as new tasks are learned. This visualization confirms that EWC efectively protects previous task performance.