a) Introduction:
Pedestrian Crossing Prediction (PCP) in driving scenes plays a critical role in ensuring the safe operation of intelligent vehicles. Due to the limited observations of pedestrian crossing behaviors in typical situations, recent studies have begun to leverage synthetic data with flexible variation to boost prediction performance, employing domain adaptation frameworks. However, different domain knowledge has distinct cross-domain distribution gaps, which necessitates suitable domain knowledge adaption ways for PCP tasks. In this work, we propose a Gated Syn-to-Real Knowledge transfer approach for PCP (Gated-S2R-PCP). Gated-S2R-PCP advocates three different pathways for transferring the pedestrian locations, visual context, and semantic-depth clues in the synthetic data to the real data that is only with the pedestrian locations and raw RGB frames.
Fig. 1: Gated-S2R-PCP advocates different knowledge transfer ways from synthetic data to real data.
A Learnable Gated Unit (LGU) is employed to fuse suitable cross-domain knowledge to boost pedestrian crossing prediction. We construct a new synthetic benchmark S2R-PCP-3181 with 3181 sequences (489,740 frames) which contains the pedestrian locations, RGB frames, semantic images, and depth images. With the synthetic S2R-PCP-3181, we transfer the knowledge to two real challenging datasets of PIE and JAAD, and superior PCP performance is obtained to the state-of-the-art methods. Additionally, we evaluate the Gated-S2R-PCP on all pedestrian crossing sequences (50 ones) in the DADA-2000 dataset to verify the PCP performance in near-collision scenes.
The flowchart of Gated-S2R-PCP is shown as follows.
Fig. 2. The flowchart of Gated-S2R-PCP.
b) Code: The code can be downloaded from here. (Extraction code:ljhq).
To run the main.py, the training phase can be conducted.
c) Dataset: The dataset S2R-PCP-3181 can be downloaded from here. (Extraction code:gkap).
Note: The whole size of S2R-PCP-3181 takes 300G space.