Testing on test sets up to current tasks

[cuongpxu]

Hi,
I'm curious about the testing phase of the base strategy. Is this necessary to test all tasks test set every time when we finish training a task? Why we do not test on all tasks up to the current training task only? Testing all task test sets seem to violate the fundamental of continual learning where you have access to the test set of the future tasks and it also slows down the learning process.

Is there any way to test on all task tests set up to the current task only?

Many thanks

[AndreaCossu]

Hi @cuongpxu , in Avalanche you are not forced to test on all tasks.

You can call strategy.eval(benchmark.test_stream[index]), where index selects the test experiences you want to evaluate on. It works like usual list indexing in Python. If you don't use index, you will select all the test experiences but that is not mandatory.
Testing on future experiences it is useful sometimes to evaluate performance on unseen tasks (like in Forward transfer, yet to be implemented in avalanche)