Performance on action recognition

[hanoonaR]

Hi Authors,

Thank you for sharing your great work. I'm curious about the performance of your models on action recognition tasks. Have you attempted to benchmark on any standard action recognition tasks such as SSV2, K400/700?

Thank you.

[klauscc]

We take TimeSformer's codebase and use our video encoder as the initialization. The results on K400 is about 80%, about 2% higher than TimeSformer with the same architecture. However, very sadly it is much lower than CLIP's vision encoder (~85% if I remember correctly). This may indicate CLIP has stronger representations but our model has better video and text alignment.

[Andy1621]

@klauscc In my previous study, directly fine-tuning CLIP-based TimeSformer achieves about 82%~83%. The similar results also can found in ST-Adapter.