VideoIntern not only recognizes known classes accurately but also has a strong perception ability for unknown classes that are out of training classes. This repo is one of the generalization tasks——Open Set Action Recognition. Specifically, we finetune VideoMAE backbone with a linear classification head on UCF-101 dataset from the evidential deep learning (EDL) perspective, without any model calibration methods used in DEAR. Our VideoIntern model achieves significant and consistent performance gains compared to multiple action recognition backbones (i.e., I3D, TSM, SlowFast, TPN), which are trained in the DEAR way, with HMDB-51 or MiT-v2 dataset as the unknown.
This repo is developed from MMAction2 codebase.
a. Create a conda virtual environment of this repo, and activate it:
conda create -n OSAR python=3.7 -y
conda activate OSARb. Install PyTorch and TorchVision following the official instructions, e.g.,
conda install pytorch=1.7.0 cudatoolkit=11.0 torchvision=0.8.0 -c pytorchc. Install mmcv, we recommend you to install the pre-build mmcv as below.
pip install mmcv-full==1.2.2 -f https://download.openmmlab.com/mmcv/dist/cu110/torch1.7.0/index.htmlImportant: If you have already installed mmcv and try to install mmcv-full, you have to uninstall mmcv first by running pip uninstall mmcv. Otherwise, there will be ModuleNotFoundError.
d. Clone the source code of this repo:
git clone https://github.com/VideoIntern/Open-Set-Action-Recognition.git Open-Set-Action-Recognition-main
cd Open-Set-Action-Recognition-maine. Install build requirements and then install OSAR.
pip install -r requirements/build.txt
python setup.py developIf no error appears in your installation steps, then you are all set!
This repo uses standard video action datasets, i.e., UCF-101 for closed set training, and HMDB-51 and MiT-v2 test sets as two different unknowns. Please refer to the default MMAction2 dataset setup steps to setup these three datasets correctly.
Note: You can just ignore the Step 3. Extract RGB and Flow in the referred setup steps since all codes related to our paper do not rely on extracted frames and optical flow. This will save you large amount of disk space!
To test our pre-trained models (see the Model Zoo), you need to download a model file and unzip it under work_dir. Let's take the I3D-based DEAR model as an example. First, download the pre-trained I3D-based models, where the full DEAR model is saved in the folder finetune_ucf101_i3d_edlnokl_avuc_debias. The following directory tree is for your reference to place the downloaded files.
work_dirs
├── mae
│ ├── finetune_ucf101_mae_edlnokl
│ │ └── latest.pth
a. Get Uncertainty Threshold. The threshold value of one model will be reported.
cd experiments/mae
# run the thresholding with BATCH_SIZE=16 on 8 GPUs
bash run_get_mae_threshold.sh edlnokl 16 8b. Out-of-Distribution Detection. The uncertainty distribution figure of a specified model will be reported.
cd experiments/mae
bash run_ood_mae_dist_detection.sh HMDB edlnokl 8c. Compute AUROC. The AUROC score of a specified model will be reported.
cd experiments/mae/results
python compute_auroc.pycd experiments/mae
bash finetune_mae_edlnokl_ucf101.sh 8The pre-trained weights (checkpoints) are available below.
| Model | Checkpoint | Train Config | Test Config | Open Set AUC (%) | Closed Set ACC (%) |
|---|---|---|---|---|---|
| I3D + DEAR | / | train | test | 77.08 / 81.54 | 93.89 |
| TSM + DEAR | / | train | test | 78.65 / 83.92 | 94.48 |
| TPN + DEAR | / | train | test | 79.23 / 81.80 | 96.30 |
| SlowFast + DEAR | / | train | test | 82.94 / 86.99 | 96.48 |
| InternVideo-B + EDL | ckpt | train | test | 83.21 / 88.98 | 96.91 |
| InternVideo-L + EDL | ckpt | train | test | 83.82 / 91.13 | 97.36 |
| InternVideo-H + EDL | ckpt | train | test | 85.48 / 91.85 | 97.89 |
For the pretrained MAE model, please download it in the Google Drive.
In addition to the MMAction2 codebase, this repo contains modified codes from:
- pytorch-classification-uncertainty: for implementation of the EDL (NeurIPS-2018).
- ARPL: for implementation of baseline method RPL (ECCV-2020).
- OSDN: for implementation of baseline method OpenMax (CVPR-2016).
- bayes-by-backprop: for implementation of the baseline method Bayesian Neural Networks (BNNs).
- rebias: for implementation of HSIC regularizer used in ReBias (ICML-2020)
We sincerely thank the owners of all these great repos!