Authors:
Hanjun Choi、Hyunsung Kim、Minho Lee、Chang-Jo Kim、Jinsung Yoon、Sang-Ki Ko
Paper:
https://arxiv.org/abs/2408.10878
Introduction
In various spatiotemporal domains such as transportation, robotics, surveillance, and sports, handling multi-agent trajectory data is crucial. However, acquiring complete trajectory data is challenging due to practical issues like signal loss in wearable devices or occlusions in computer vision systems. This problem is particularly pronounced in sports, where players can disappear from the camera view. To address the prevalence of missing values in multi-agent trajectory data, this paper proposes a Derivative-Based Hybrid Prediction (DBHP) framework. This framework aims to impute missing trajectories accurately by leveraging physical constraints and derivative information.
Related Work
Several studies have addressed the imputation of time series data, but few have tackled the specific challenges of multi-agent trajectories with complex interactions. Methods like TimesNet and TIDER focus on multivariate time series but are unsuitable for multi-agent spatiotemporal domains. Other studies, such as SPIN, assume fixed sensor locations, making them inapplicable to moving agents. Recent works like BRITS, NAOMI, and NRTSI have attempted to model agents’ interactions and dynamics but face limitations in handling scenarios with varying missing periods or compounding errors. The proposed DBHP framework aims to overcome these limitations by integrating direct predictions and derivative-accumulating predictions.
Research Methodology
Neural Network-Based Direct Prediction
The DBHP framework starts with a neural network equipped with Set Transformers and bidirectional LSTMs to produce naive predictions of missing trajectories. The Set Transformer ensures permutation-equivariance, meaning the order of input agents does not affect the output. The neural network generates direct predictions (DP) for each agent’s position, velocity, and acceleration.
Derivative-Accumulating Prediction
To enhance the accuracy and smoothness of predictions, the framework also computes Derivative-Accumulating Predictions (DAPs). These predictions accumulate predicted velocities and accelerations, enforcing physical relationships between position, velocity, and acceleration. This approach helps produce more stable and realistic trajectories.
Dynamic Hybrid Prediction
The final step involves combining the direct predictions and DAPs using a weighted sum. An additional Bi-LSTM is used to determine the weights dynamically, ensuring the final prediction leverages the strengths of both DP and DAP. The framework minimizes a loss function that includes terms for both direct and derivative-accumulating predictions to ensure overall accuracy.
Experimental Design
Data Preparation
The experiments were conducted on three public datasets from soccer, basketball, and American football. The datasets were downsampled to a common rate of 10Hz for consistency. Different missing scenarios were simulated, including uniform missing, agent-wise missing, and broadcasting camera scenarios.
Missing Scenarios
- Uniform Missing Scenario: All players have missing values at the same time interval.
- Agent-wise Missing Scenario: Individual players have missing values at different time intervals.
- Broadcasting Camera Scenario: A virtual camera follows the ball, capturing only players inside the camera view.
Models and Evaluation Metrics
The performance of the DBHP framework was compared with several baseline models using two metrics:
– Position Error (PE): The average Euclidean distance between the true and predicted positions.
– Step Change Error (SCE): The average absolute difference between the variance of the true and predicted velocities.
Results and Analysis
Performance Comparison
The DBHP framework significantly outperformed other baselines across different datasets and missing scenarios. The inclusion of permutation-equivariance and derivative information was crucial for achieving higher accuracy. The framework also demonstrated data efficiency, performing well even with limited training data.
Ablation Studies
Ablation studies showed the importance of using derivative features in both DP and DAP. Models that only observed positional information suffered from significant performance degradation. The framework’s performance was robust against varying window sizes and missing rates, particularly in the broadcasting camera scenario.
Practical Applications
The DBHP framework can be applied to practical tasks such as pitch control analysis and approximating match statistics from incomplete tracking data. The framework provided accurate estimates of player statistics, demonstrating its utility in real-world scenarios.
Overall Conclusion
The DBHP framework effectively imputes missing values in multi-agent trajectory data by integrating direct predictions and derivative-accumulating predictions. The experiments demonstrated its superior performance in terms of accuracy and realism. The framework’s ability to generalize across various sports and its applicability to practical tasks make it a valuable tool for handling incomplete trajectory data in spatiotemporal domains. Future work will explore its application to other domains and practical tasks requiring complete trajectories of multiple agents.