Making sure the Evaluation happens on GPU

[DannyFGitHub]

I've passed the cuda device to every strategy I'm using at the point of defining the strategy e.g.:

DEVICE = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

gdumb_strategy = GDumb(
            model, optimizer, prepare_criterion(),
            train_mb_size=TRAIN_MINIBATCH_SIZE, train_epochs=epochs_num, eval_mb_size=EVAL_MINIBATCH_SIZE,
            evaluator=prepare_evaluator(scenario, model_description_name + "_GDUMB", class_names),
            plugins=plugins,
            eval_every=EVAL_EVERY,
            device=DEVICE
        )

I've noticed that training happens at the rate of GPU. 100it/s+

Epoch 28 ended.
100%|██████████| 75/75 [00:00<00:00, 128.49it/s]

But evaluation happens at the rate of the cpu 4.58it/s+

-- >> Start of eval phase << --
-- Starting eval on experience 1 (Task 0) from train stream --
100%|██████████| 75/75 [00:16<00:00,  4.58it/s]

Whats the avalanche way of making sure that evaluation also runs through gpu for best speed?
(Also minor side question, might be an easy one, why does avalanche start with evaluating, then moves onto training?)

[vlomonaco]

Hi @DannyFGitHub! I believe the gpu/cpu usage depends on how the train / eval plugins are implemented. It may be reasonable to have some eval metrics work on cpu only since it does not really make a difference in terms of performance.

why does avalanche start with evaluating, then moves onto training?

For some metrics it may be interesting to record their value before starting training (e.g. RAM usage)

[DannyFGitHub]

That helps, knowing that, I performed split half on my metrics disabling until I isolated down to the disk_usage_metrics making the eval happen slower. When I turn it off it evals faster.
e.g.

eval_plugin = EvaluationPlugin(
        accuracy_metrics(minibatch=False, epoch=False, experience=True, stream=False),
        loss_metrics(minibatch=False, epoch=False, experience=True, stream=False),
        timing_metrics(experience=True, epoch=False),
        forgetting_metrics(experience=True, stream=False),
        bwt_metrics(experience=True, stream=False),
        cpu_usage_metrics(experience=True),
        confusion_matrix_metrics(image_creator=Confusion_Matrix_Maker(class_names).custom_cm_image_creator,
                                 class_names=class_names,
                                 num_classes=scenario.n_classes, save_image=True, stream=False),
        # disk_usage_metrics(minibatch=False, epoch=False, experience=True, stream=False), <------ Commented out
        collect_all=True,
        benchmark=scenario,
        loggers=[interactive_logger, text_logger, tb_logger],
    )

[AntonioCarta]

So, disk_usage_metrics are making the eval run much slower? Have you checked that eval happens on the GPU? This could be a bug, or at least we can try to optimize it.

[DannyFGitHub]

I found that with disk_usage_metrics both eval and training were slowed down, and without they were both faster.

I found an easy way to recreate this using the tutorial code.

My Setup:
NVidia RTX 3080 Ti
Cuda 11.5

This is exactly what I did:
0) Create virtual-env:
python3 -m venv venv; . ./venv/bin/activate

1. Install avalanche:
   pip install avalanche-lib

2. Install pytorch with cuda:
   pip3 install torch==1.10.1+cu113 torchvision==0.11.2+cu113 torchaudio==0.10.1+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html

3. Create a main.py file and add this code in it.
   Reference: https://avalanche.continualai.org/from-zero-to-hero-tutorial/07_putting-all-together

from torch.optim import SGD
from torch.nn import CrossEntropyLoss
from avalanche.benchmarks.classic import SplitMNIST
from avalanche.evaluation.metrics import forgetting_metrics, accuracy_metrics, \
    loss_metrics, timing_metrics, cpu_usage_metrics, confusion_matrix_metrics, disk_usage_metrics
from avalanche.models import SimpleMLP
from avalanche.logging import InteractiveLogger, TextLogger, TensorboardLogger
from avalanche.training.plugins import EvaluationPlugin
from avalanche.training.strategies import Naive

scenario = SplitMNIST(n_experiences=5)

# MODEL CREATION
model = SimpleMLP(num_classes=scenario.n_classes)

# DEFINE THE EVALUATION PLUGIN and LOGGERS
# The evaluation plugin manages the metrics computation.
# It takes as argument a list of metrics, collectes their results and returns
# them to the strategy it is attached to.

# log to Tensorboard
tb_logger = TensorboardLogger()

# log to text file
text_logger = TextLogger(open('log.txt', 'a'))

# print to stdout
interactive_logger = InteractiveLogger()

eval_plugin = EvaluationPlugin(
    accuracy_metrics(minibatch=True, epoch=True, experience=True, stream=True),
    loss_metrics(minibatch=True, epoch=True, experience=True, stream=True),
    timing_metrics(epoch=True, epoch_running=True),
    forgetting_metrics(experience=True, stream=True),
    cpu_usage_metrics(experience=True),
    confusion_matrix_metrics(num_classes=scenario.n_classes, save_image=False,
                             stream=True),
    disk_usage_metrics(minibatch=True, epoch=True, experience=True, stream=True),
    loggers=[interactive_logger, text_logger, tb_logger]
)

# CREATE THE STRATEGY INSTANCE (NAIVE)
cl_strategy = Naive(
    model, SGD(model.parameters(), lr=0.001, momentum=0.9),
    CrossEntropyLoss(), train_mb_size=500, train_epochs=1, eval_mb_size=100,
    evaluator=eval_plugin)

# TRAINING LOOP
print('Starting experiment...')
results = []
for experience in scenario.train_stream:
    print("Start of experience: ", experience.current_experience)
    print("Current Classes: ", experience.classes_in_this_experience)

    # train returns a dictionary which contains all the metric values
    res = cl_strategy.train(experience)
    print('Training completed')

    print('Computing accuracy on the whole test set')
    # test also returns a dictionary which contains all the metric values
    results.append(cl_strategy.eval(scenario.test_stream))

4. Add the cuda device to the Naive strategy. e.g. device="cuda"

Then run with the disk_usage_metrics(minibatch..... line and then without the disk_usage_metrics line.
Without it runtime: 0:00:13.773714 (14 secs) - ~ 90 it/s
With the disk usage metric it runtime: 0:02:03.999363 (2 minutes) ~ 5 it/s

Also tested by deleting the project folder, and following steps above skipping step 2, and specifying device="cpu" and removing disk_usage_metrics runtime was: 0:00:14.228788 (14 secs)
With disk_usage_metrics runtime was: 0:01:43.784600 (1 minute 43 secs) ~ 6 it/s

(I think CPU and GPU are in par here due to the type of CL work, for my other avalanche project GPU makes a big difference in reducing run time)

P.S. For my graphics card if you skip step 2 and try to set device="cuda" then this error will appear:

NVIDIA GeForce RTX 3080 Ti with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
If you want to use the NVIDIA GeForce RTX 3080 Ti GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/
....
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

Montoring with nvtop

While in avalanche training/eval session with disk_usage_metrics commented out, and device='cuda' shows 12% gpu utilisation, while on device="cpu" it sits around 1-3% gpu utilisation. With disk_usage_metrics active, device="cuda" shows approx 1% gpu utilisation.

[AntonioCarta]

NVIDIA GeForce RTX 3080 Ti with CUDA capability sm_86 is not compatible with the current PyTorch installation. The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70. If you want to use the NVIDIA GeForce RTX 3080 Ti GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/ .... RuntimeError: CUDA error: no kernel image is available for execution on the device CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

This looks like an error with your pytorch installation. With this error, pytorch can't use the GPU.

[DannyFGitHub]

You are right, however, I'm not too concerned about that error, I'm only showing the error that happens if you skip step 2 in my steps (providing rationale for step 2). Because of that error you have to do step 2 and install the pytorch cuda libraries.

2. Install pytorch with cuda:
pip3 install torch==1.10.1+cu113 torchvision==0.11.2+cu113 torchaudio==0.10.1+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html

If you install pytorch with cuda as detailed in pytorch site then the error goes away. Just showing for rationale as to what step 2 is for.