Filter Deform Attention GAN: Constructing Human Motion Videos from Few Images

Our image warping-based method implements a highly robust motion transmission model, Filter Deform Attention Generative Adversarial Network (FDA GAN), which can transmit complex human motion videos using only a small number of human body images. We use a 3D pose shape estimator to replace traditional 2D pose estimation. We design a new attention mechanism and fuse it with the GAN network to propose a new network that can extract image features well and generate human motion videos. The entire process of FDA GAN is divided into three parts. Firstly, we obtain the pose and shape of the human through the 3D pose shape estimator. Secondly, we analyze the pose shape to obtain the foreground and background images of the human, as well as motion transformation streams. Finally, in the generative adversarial network, the foreground, background, and texture features are fused into new-person images. We show in the experimental section that our method outperforms recent methods overall and on various evaluation metrics.

Approach

IPER datasets

https://svip-lab.github.io/dataset/iPER_dataset.html

Installation

Environment Setup

This repository has been tested on the following platform: Python 3.8.15, PyTorch 1.7.0 with CUDA 11.0 and cuDNN 8.0, Ubuntu 22.04

To clone the repo, run either:

git clone --recursive https://github.com/mioyeah/FDA-GAN.git

Next, you have to make sure that you have all dependencies in place. The simplest way to do so, is to use anaconda.

conda env create -f environment.yml -n my-env
conda activate my-env

Pre-trained Models

Download the following models and place them in their correspoding directories:

• FDAGAN.pth in assets/checkpoints/neural_reders.

Inference

python motiontransfer.py \
      --gpu_ids 0 \
      --image_size 512 \
      --num_source 2 \
      --output_dir "./results" \
      --assets_dir "./assets" \
      --model_id "Av37667655_2" \
      --src_path "path?=./assets/samples/sources/fange_1/fange_1_ns=6,name?=fange_1_ns=6" \
      --ref_path "path?=./assets/samples/references/Av37667655_2.mp4,name?=Av37667655_2,pose_fc?=300"