Task-accuracy changing based on 'return_task_id' property

[Mattdl]

I need task labels for a custom task-specific metric I defined.
To get the separate task-labels I define "return_task_id=True" in the SplitMNIST benchmark.

However, this also changes the way the accuracies are calculated.

• e.g. with return_task_id=True: Finetuning on the second task in Split-MNIST gives about 50% ACC on the first task (
• while for return_task_id=False: Finetuning on the second task in Split-MNIST gives 0% ACC on the first task

I want to maintain the results where ACC as in the return_task_id=False setup, even if I do require task labels for other metrics with return_task_id=True. Is this possible?

[vlomonaco]

Hi @Mattdl! What accuracy metrics are you referring to? Can you send us a minimal snippet of code to reproduce the issue? @AndreaCossu implemented this and may have additional insights!

[Mattdl]

Thanks for the response! I'm having the issue with the ExperienceAccuracy (but could also have effect on the others, didn't test that).
You can try it out on the examples/all_mnist.py and add the ExperienceAccuracy. The two test cases are with return_task_id=True and False.

[AntonioCarta]

Wait. Are you sure that this is a metric problem and not the labels? If I remember correctly, by default task_id=True also changes the labels. In the second task you will have labels 0 and 1 (instead of different ones for each task). If you have a single head, that's why you get 50% in a multitask instead of 0%.

[Mattdl]

I think you're right, that could be the cause! I had been playing around with the Accuracy metrics, but couldn't figure out why I kept getting predictions/labels of 0,1 (from the first task). I'm using the single head setting indeed. In that case, it's probably a bug in the ExperienceAccuracy then?

[AntonioCarta]

It's not a bug, it's a default behavior which is a bit counterintuitive. It sets class_ids_from_zero_in_each_exp = True in the nc_benchmark builder

[AntonioCarta]

Maybe we should change the behavior here. I don't think anyone expects the benchmarks to change the output space by default.

[Mattdl]

I see, thanks! I would also opt to not let the output space/metrics depend on the benchmark definition.