Skip to content

Efficient Adversarial Contrastive Learning via Robustness-aware Coreset Selection (NeurIPS 2023 Spotlight)

Notifications You must be signed in to change notification settings

GodXuxilie/Efficient_ACL_via_RCS

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

71 Commits
 
 
 
 
 
 
 
 

Repository files navigation

Efficient Adversarial Contrastive Learning via Robustness-aware Coreset Selection

This repository provides codes for NeurIPS 2023 (Spotlight) paper: Efficient Adversarial Contrastive Learning via Robustness-aware Coreset Selection (https://arxiv.org/pdf/2302.03857.pdf)
Xilie Xu* (NUS), Jingfeng Zhang* (RIKEN-AIP/The University of Auckland), Feng Liu (The University of Melbourne), Masashi Sugiyama (RIKEN-AIP/The University of Toyko), Mohan Kankanhalli (NUS).


Robustness-aware coreset selection (RCS) can significantly speed up both robust self-supervised pre-training and supervised pre-training while maintaining the transferability of the pre-trained models.

RCS outputs a coreset which is an informative training subset. The model can achieve the minimized representational divergence between natural data and their adversarial counterparts after training on the coreset selected by RCS. In this way, RCS helps maintain the adversarial robustness of representations while accelerating robust pre-training.

Script

In our repo, we provide the codes and scripts of RCS for speeding up robust pre-training as follows:

  • Efficient adversarial contrastive learning (ACL) via on unlabeled datasets (CIFAR-10/CIFAR-100/STL-10/ImageNet-1K);
  • Efficient standard adversarial training (SAT) via RCS on labelled datasets (ImageNet-1K).
cd ./ACL_RCS/small_scale_datasets
python DynACL_RCS.py exp_dir --ACL_DS --dataset dataset --fraction 0.2
cd ./ACL_RCS/ImageNet_32
python ACL_RCS.py exp_dir --ACL_DS --fraction 0.05

As for preparing ImageNet-1K of $32 \times 32$ resolution, we use the following scripts:

wget https://image-net.org/data/downsample/Imagenet32_train.zip
wget https://image-net.org/data/downsample/Imagenet32_val.zip
cd ./SAT_RCS/ImageNet_32
python SAT_RCS.py --out_dir exp_dir --fraction 0.2
cd ./SAT_RCS/ImageNet_224
python SAT_RCS.py --out_dir exp_dir --fraction 0.2

As for preparing ImageNet-1K of $224\times 224$ resolution, we use the following scripts:

wget https://image-net.org/data/ILSVRC/2012/ILSVRC2012_img_train.tar
wget https://image-net.org/data/ILSVRC/2012/ILSVRC2012_img_val.tar
wget https://image-net.org/data/ILSVRC/2012/ILSVRC2012_devkit_t12.tar.gz

Pre-Trained Checkpoints

We provide the following pre-trained checkpoints:

  • Efficient ACL via RCS on CIFAR-10/CIFAR-100/STL-10 using ResNet-18 in Model Zoo of RobustSSL Benchmark;
  • Efficient ACL via RCS on ImageNet-1K of $32 \times 32$ resolution using WRN-28-10 in Dropbox;
  • Efficient SAT via RCS on ImageNet-1K of $32 \times 32$ resolution using WRN-28-10 in Dropbox;
  • Efficient SAT via RCS on ImageNet-1K of $224 \times 224$ resolution using ResNet-50 in Dropbox.

Package

  • Python 3.8
  • Pytorch 1.13
  • CUDA 11.6

BibTeX

@inproceedings{
xu2023efficient,
title={Efficient Adversarial Contrastive Learning via Robustness-Aware Coreset Selection},
author={Xilie Xu and Jingfeng Zhang and Feng Liu and Masashi Sugiyama and Mohan Kankanhalli},
booktitle={Thirty-seventh Conference on Neural Information Processing Systems},
year={2023},
url={https://openreview.net/forum?id=fpzA8uRA95}
}

Contact

Please drop an e-mail to [email protected] and [email protected] if you have any issue.

About

Efficient Adversarial Contrastive Learning via Robustness-aware Coreset Selection (NeurIPS 2023 Spotlight)

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published