[Theory] Determining the Type of Scenario

[akashtadwai]

Hi all, I have a doubt identifying the type of Scenario for one of the experiments. I have a Network Intrusion dataset, it contains 15 classes (1 Normal and 14 attacks). But I'm only interested in whether they are of a type normal or attack and hence, I have changed the labels of data such that the normal class has a label of 0 and all the remaining attack classes have labeled as 1. I then created a benchmark using avalanche's tensor_scenario. I have divided the dataset into 5 tasks each containing 3 classes in the original dataset. For example, this dictionary contains the division of the tasks, class 0 has a label 0 and all remaining classes have a label 1.

{'1': [0,1,2], '2': [3,4,5], '3': [6,7,8], '4': [9,10,11], '5': [12,13,14]}

I am unclear about the following questions in this experiment.

1. How does tensor_scenario make use of task labels during test time, i.e, does it provide task labels during test time as in Inc. Task learning OR The task_labels attribute is to just number the sequence of tasks and they don't provide the taskID during the test time.
2. Under which Scenarios of CL does this type of setting come under? (Domain Inc or Class Inc). I'm confused because, in Domain Inc. case new examples of the same classes(labels) are encountered over time and the output space of all the tasks is the same (section 2 of Re-evaluating CL scenarios), which is not the case here because output space is not the same in all tasks (one of the tasks has 2 labels (both 0 and 1 labels are present in the task1 or above example order) and all others have only one label (others belong to attack class hence have a label 1) ). And it's not class-incremental(?) because I have different instances of the same label (attack) distributed across tasks. (The Original data distribution of attack classes is different, but all of them have the same label).

I need some suggestions regarding what kind of scenario does this type of task division comes under.

[GMvandeVen]

Hi Akash, that’s an interesting problem! I would say that the problem you describe is a mixture of domain-incremental learning and class-incremental learning. I think a useful way to look at it is that there are two “incremental axes” in your task set: (1) the first incremental axis is at the level of the labels, as the first task contains the labels ‘0’ and ‘1’, while the other tasks only contain the label ‘1’ (this incremental aspect of your data makes that your problem is partly class-incremental); and (2) the second incremental axis is at the level of the inputs, as the input distribution corresponding to label ‘1’ changes between tasks (this incremental aspect of your data makes that your problem is partly domain-incremental).

[GMvandeVen]

Regarding the implementation in Avalanche, the way you describe your implementation sounds reasonable to me, although it might be good if someone with more experience with creating custom datasets in Avalanche could confirm this.
My understanding is that Avalanche will always provide you with the task_labels, also at test time, so you would have to make sure yourself that they are not used, at least not at test time. (From your problem description it is not clear whether you assume knowledge of the task_labels during training, but either way could work.)

[akashtadwai]

@GMvandeVen I want task_labels not to be used during the test time as in the case of Domain and Class Incremental learning. I have defined the tensor_scenario as

generic_scenario = tensor_scenario(
    train_data_x=dataset[0],
    train_data_y=dataset[1],
    test_data_x=dataset[2],
    test_data_y=dataset[3],
    task_labels=[int(key)-1 for key in perm.keys()]
)

Does this give access to the task labels during test time as in the case of Class incremental learning or is the tasks_labels attribute is just to number the sequence of tasks, where dataset[i] is a list of tensors.

[GMvandeVen]

My understanding is that with the implementation you describe here, Avalanche will provide the model with the task_labels both during training and during testing. (You would thus have to make sure yourself that the model does not use the task_labels during testing.)

To prevent Avalanche from providing the model with the task_labels, you could instead input a list of zeros as task_labels:

generic_scenario = tensor_scenario(
    train_data_x=dataset[0],
    train_data_y=dataset[1],
    test_data_x=dataset[2],
    test_data_y=dataset[3],
    task_labels=[0 for key in perm.keys()]
)

Note that this means that also during training, the model will not be provided with the informative task_labels.

As far as I’m aware, at the moment it is not possible to specify a benchmark in such a way that the task_labels are provided during training but not during testing; perhaps it would be good for such a feature to be added to Avalanche.

[vlomonaco]

@akashtadwai marking this question as solved, please let me know in case you want to re-open it! :)