ONSEP: A Novel Online Neural-Symbolic Framework for Event Prediction Based on Large Language Model

Authors:

Xuanqing Yu、Wangtao Sun、Jingwei Li、Kang Liu、Chengbao Liu、Jie Tan

Paper:

https://arxiv.org/abs/2408.07840

Introduction

Event prediction is a critical area of research due to its potential to minimize losses associated with future events. Temporal Knowledge Graph Forecasting (TKGF) is a technique used to model real-world event data, representing complex interactions between entities over time. Traditional approaches to TKGF involve converting event data into Temporal Knowledge Graphs (TKGs) and using Graph Neural Networks (GNNs) and Recurrent Neural Networks (RNNs) to capture evolving entity relationships. However, these methods are resource-intensive and often fail to capture long-term trends and causal relationships effectively.

To address these limitations, the Online Neural-Symbolic Event Prediction (ONSEP) framework is introduced. ONSEP integrates Dynamic Causal Rule Mining (DCRM) and Dual History Augmented Generation (DHAG) to enhance event prediction accuracy and adaptability. This framework leverages the understanding and generative capabilities of Large Language Models (LLMs) without requiring extensive retraining.

Temporal Knowledge Graph Forecasting

A Temporal Knowledge Graph (TKG) is a time-sequenced series of multi-relational directed graphs. Each snapshot of the graph at time ( t ) is represented as ( G_t = (V, R, E_t) ), where ( V ) denotes entities, ( R ) denotes relations, and ( E_t ) comprises timestamped facts as quadruples ((s, r, o, t)). TKGF aims to predict future states of the graph by inferring missing entities in a quadruple for a future time.

ICL for Temporal Knowledge Graph Forecasting

In-Context Learning (ICL) enables LLMs to adapt to new tasks through contextual examples without fine-tuning. For TKGF, ICL leverages historical data to generate predictions for future events. However, ICL may not fully capture long-term trends and causal relationships due to limitations in the length of history of LLM inputs.

Causal Rules in TKG

Causal rules in TKG capture cause-and-effect links between events. A causal rule base (CRB) is a set of tuples, each comprising a causal rule and its confidence score. The CRB dynamically updates to reflect new causal relationships, enhancing the adaptability of the event prediction model.

Method

The ONSEP framework consists of two key modules: Dynamic Causal Rule Mining (DCRM) and Dual History Augmented Generation (DHAG).

Dynamic Causal Rule Mining

DCRM identifies causal relationships within the TKG using a semantic-driven rule learning algorithm. This phase includes:

1. Semantic-Driven Rule Learning: This module filters candidate causes, assesses causality using LLMs, and constructs causal rules.
2. Dynamic Update: The causal rule base is dynamically updated with new rules and confidence levels.
3. Rule Filter and Sort by Confidence: Rules below a predefined confidence threshold are filtered out, and the remaining rules are sorted by confidence.

Dual History Augmented Generation

DHAG integrates short-term and long-term historical contexts to enhance event prediction accuracy. It includes:

1. Long Short-Term Bi-Branch Retriever: This component retrieves short-term and long-term historical event chains.
2. Hybrid Model Inference: This strategy merges the query with short-term and long-term historical contexts, producing distinct probabilities for each context and combining them using a weighted ensemble approach.

Experiments

Experimental Settings

The ONSEP framework was evaluated on subsets of the Integrated Crisis Early Warning System (ICEWS) dataset, including ICEWS14, ICEWS05-15, and ICEWS18. These datasets contain timestamped records of political events, making them suitable for temporal analysis.

Performance Comparison

ONSEP demonstrated significant performance improvements over the ICL method across all datasets. Specifically, ONSEP achieved notable Hit@1 improvements, indicating enhanced precision in event prediction.

Inductive Setting and the Effectiveness of DCRM

Experiments showed that causal rules mined during test-time iterations are transferable across datasets with similar relational structures. Incorporating DCRM led to enhanced performance, highlighting its effectiveness in improving inference accuracy and recall.

Effectiveness of DHAG

The DHAG module’s impact was assessed by varying the weighted fusion ratio. The optimal performance was achieved with a balanced approach, indicating the importance of integrating both short-term and long-term historical contexts.

Performance Comparison of Different Model Scale and Series

Model performance improved with increased parameter scale, with the 20B models outperforming the 7B models. ONSEP enhanced performance across different model scales and series, demonstrating its adaptability and effectiveness.

Hyperparameter Sensitivity Analysis

Hyperparameter sensitivity analysis highlighted the importance of selecting the right history length, rule selection thresholds, and fusion ratios for optimal model performance. Detailed discussions on these findings are provided in the appendix.

Conclusion

The ONSEP framework integrates LLMs with TKGs to achieve adaptive and precise event forecasting in dynamic online environments. By addressing the limitations of traditional methods, ONSEP demonstrates significant performance improvements across various datasets. This framework shows great potential for applications in financial forecasting, public sentiment monitoring, and recommendation systems.

Limitations

The ONSEP framework requires multiple uses of large models, leading to increased inference time and higher computational costs. The effectiveness of the model is influenced by the length of the input context, and the method faces interpretability challenges due to unclear reasoning paths.

Ethics Statement

This research was conducted using publicly available datasets, ensuring no data privacy concerns. The study does not involve human annotations, and there are no related ethical concerns.

Acknowledgements

This work was supported by the National Key Research and Development Program of China and the National Nature Science Foundation of China.

By integrating dynamic causal rule mining and dual history augmented generation, the ONSEP framework represents a significant advancement in the field of temporal knowledge graph forecasting, offering a robust and adaptable solution for diverse event prediction tasks.

Datasets:

ICEWS