Skip to content

Commit

Permalink
Update gitbook documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
ContinualAI-bot committed Dec 13, 2022
1 parent 7c331c6 commit 5166414
Show file tree
Hide file tree
Showing 4 changed files with 294 additions and 50 deletions.
10 changes: 6 additions & 4 deletions docs/gitbook/how-tos/avalanchedataset/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,14 @@ description: Dealing with AvalancheDatasets

# AvalancheDataset

The `AvalancheDataset` is an implementation of the PyTorch `Dataset` class that comes with many useful out-of-the-box functionalities. For most users, the _AvalancheDataset_ can be used as a plain PyTorch Dataset. For classification problems, `AvalancheDataset` return `x, y, t` elements (input, target, task label). However, the `AvalancheDataset` can be easily extended for any custom needs.
The `AvalancheDataset` is an implementation of the PyTorch `Dataset` class that comes with many useful out-of-the-box functionalities. For most users, the *AvalancheDataset* can be used as a plain PyTorch Dataset. For classification problems, `AvalancheDataset` return `x, y, t` elements (input, target, task label). However, the `AvalancheDataset` can be easily extended for any custom needs.

**A serie of **_**Mini How-Tos**_ will guide you through the functionalities of the _AvalancheDataset_ and its subclasses:
**A serie of _Mini How-Tos_** will guide you through the functionalities of the *AvalancheDataset* and its subclasses:

- [AvalancheDatasets basics](https://avalanche.continualai.org/how-tos/avalanchedataset/avalanche-datasets)
- [Advanced Transformations](https://avalanche.continualai.org/how-tos/avalanchedataset/advanced-transformations)

* [AvalancheDatasets basics](https://avalanche.continualai.org/how-tos/avalanchedataset/avalanche-datasets)
* [Advanced Transformations](https://avalanche.continualai.org/how-tos/avalanchedataset/advanced-transformations)

```python

```
50 changes: 24 additions & 26 deletions docs/gitbook/how-tos/avalanchedataset/avalanche-datasets.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,47 +2,45 @@
description: Converting PyTorch Datasets to Avalanche Dataset
---

# avalanche-datasets

# Avalanche Datasets
Datasets are a fundamental data structure for continual learning. Unlike offline training, in continual learning we often need to manipulate datasets to create streams, benchmarks, or to manage replay buffers. High-level utilities and predefined benchmarks already take care of the details for you, but you can easily manipulate the data yourself if you need to. These how-to will explain:

1. PyTorch datasets and data loading
2. How to instantiate Avalanche Datasets
3. AvalancheDataset features

In Avalanche, the `AvalancheDataset` is everywhere:
- The dataset carried by the `experience.dataset` field is always an *AvalancheDataset*.
- Many benchmark creation functions accept *AvalancheDataset*s to create benchmarks.
- Avalanche benchmarks are created by manipulating *AvalancheDataset*s.
- Replay buffers also use `AvalancheDataset` to easily concanate data and handle transformations.

* The dataset carried by the `experience.dataset` field is always an _AvalancheDataset_.
* Many benchmark creation functions accept _AvalancheDataset_s to create benchmarks.
* Avalanche benchmarks are created by manipulating _AvalancheDataset_s.
* Replay buffers also use `AvalancheDataset` to easily concanate data and handle transformations.

## 📚 PyTorch Dataset: general definition

In PyTorch, **a `Dataset` is a class** exposing two methods:

* `__len__()`, which returns the amount of instances in the dataset (as an `int`).
* `__getitem__(idx)`, which returns the data point at index `idx`.
- `__len__()`, which returns the amount of instances in the dataset (as an `int`).
- `__getitem__(idx)`, which returns the data point at index `idx`.

In other words, a Dataset instance is just an object for which, similarly to a list, one can simply:

* Obtain its length using the Python `len(dataset)` function.
* Obtain a single data point using the `x, y = dataset[idx]` syntax.
- Obtain its length using the Python `len(dataset)` function.
- Obtain a single data point using the `x, y = dataset[idx]` syntax.

The content of the dataset can be either loaded in memory when the dataset is instantiated (like the torchvision MNIST dataset does) or, for big datasets like ImageNet, the content is kept on disk, with the dataset keeping the list of files in an internal field. In this case, data is loaded from the storage on-the-fly when `__getitem__(idx)` is called. The way those things are managed is specific to each dataset implementation.

### Quick note on the IterableDataset class

A variation of the standard `Dataset` exist in PyTorch: the [IterableDataset](https://pytorch.org/docs/stable/data.html#iterable-style-datasets). When using an `IterableDataset`, one can load the data points in a sequential way only (by using a tape-alike approach). The `dataset[idx]` syntax and `len(dataset)` function are not allowed. **Avalanche does NOT support `IterableDataset`s.** You shouldn't worry about this because, realistically, you will never encounter such datasets (at least in torchvision). If you need `IterableDataset` let us know and we will consider adding support for them.

## How to Create an AvalancheDataset

## How to Create an AvalancheDataset
To create an `AvalancheDataset` from a PyTorch you only need to pass the original data to the constructor as follows


```python
!pip install avalanche-lib
```


```python
import torch
from torch.utils.data.dataset import TensorDataset
Expand All @@ -60,6 +58,7 @@ avl_data = AvalancheDataset(torch_data)

The dataset is equivalent to the original one:


```python
print(torch_data[0])
print(avl_data[0])
Expand All @@ -70,13 +69,13 @@ print(avl_data[0])
most of the time, you can also use one of the utility function in [benchmark utils](https://avalanche-api.continualai.org/en/latest/benchmarks.html#utils-data-loading-and-avalanchedataset) that also add attributes such as class and task labels to the dataset. For example, you can create a classification dataset using `make_classification_dataset`.

Classification dataset

* returns triplets of the form \<x, y, t>, where t is the task label (which defaults to 0).
* The wrapped dataset must contain a valid **targets** field.
- returns triplets of the form <x, y, t>, where t is the task label (which defaults to 0).
- The wrapped dataset must contain a valid **targets** field.

Avalanche provides some utility functions to create supervised classification datasets such as:
- `make_tensor_classification_dataset` for tensor datasets
all of these will automatically create the `targets` and `targets_task_labels` attributes.

* `make_tensor_classification_dataset` for tensor datasets all of these will automatically create the `targets` and `targets_task_labels` attributes.

```python
from avalanche.benchmarks.utils import make_classification_dataset
Expand All @@ -90,9 +89,9 @@ sup_data = make_classification_dataset(torch_data, task_labels=tls)
```

## DataLoader

Avalanche provides some [custom dataloaders](https://avalanche-api.continualai.org/en/latest/benchmarks.html#utils-data-loading-and-avalanchedataset) to sample in a task-balanced way or to balance the replay buffer and current data, but you can also use the standard pytorch `DataLoader`.


```python
from torch.utils.data.dataloader import DataLoader

Expand All @@ -105,9 +104,9 @@ for x_minibatch, y_minibatch in my_dataloader:
```

## Dataset Operations: Concatenation and SubSampling

While PyTorch provides two different classes for concatenation and subsampling (`ConcatDataset` and `Subset`), Avalanche implements them as dataset methods. These operations return a new dataset, leaving the original one unchanged.


```python
cat_data = avl_data.concat(avl_data)
print(len(cat_data)) # 100 + 100 = 200
Expand All @@ -119,8 +118,9 @@ print(len(avl_data)) # 100, original data stays the same
```

## Dataset Attributes
AvalancheDataset allows to add attributes to datasets. Attributes are named arrays that carry some information that is propagated by concatenation and subsampling operations.
For example, classification datasets use this functionality to manage class and task labels.

AvalancheDataset allows to add attributes to datasets. Attributes are named arrays that carry some information that is propagated by concatenation and subsampling operations. For example, classification datasets use this functionality to manage class and task labels.

```python
tls = [0 for _ in range(100)] # one task label for each sample
Expand All @@ -142,16 +142,14 @@ print(cat_data.targets_task_labels.name, len(cat_data.targets_task_labels._data)
Thanks to `DataAttribute`s, you can freely operate on your data (e.g. to manage a replay buffer) without losing class or task labels. This makes it easy to manage multi-task datasets or to balance datasets by class.

## Transformations

Most datasets from the _torchvision_ libraries (as well as datasets found "in the wild") allow for a `transformation` function to be passed to the dataset constructor. The support for transformations is not mandatory for a dataset, but it is quite common to support them. The transformation is used to process the X value of a data point before returning it. This is used to normalize values, apply augmentations, etcetera.
Most datasets from the *torchvision* libraries (as well as datasets found "in the wild") allow for a `transformation` function to be passed to the dataset constructor. The support for transformations is not mandatory for a dataset, but it is quite common to support them. The transformation is used to process the X value of a data point before returning it. This is used to normalize values, apply augmentations, etcetera.

`AvalancheDataset` implements a very rich and powerful set of functionalities for managing transformation. You can learn more about it in the [Advanced Transformations How-To](https://avalanche.continualai.org/how-tos/avalanchedataset/advanced-transformations).

## Next steps
With these notions in mind, you can start start your journey on understanding the functionalities offered by the AvalancheDatasets by going through the *Mini How-To*s.

With these notions in mind, you can start start your journey on understanding the functionalities offered by the AvalancheDatasets by going through the _Mini How-To_s.

Please refer to the [list of the _Mini How-To_s regarding AvalancheDatasets](https://avalanche.continualai.org/how-tos/avalanchedataset) for a complete list. It is recommended to start with the **"Creating AvalancheDatasets"** _Mini How-To_.
Please refer to the [list of the *Mini How-To*s regarding AvalancheDatasets](https://avalanche.continualai.org/how-tos/avalanchedataset) for a complete list. It is recommended to start with the **"Creating AvalancheDatasets"** *Mini How-To*.

## 🤝 Run it on Google Colab

Expand Down
Loading

0 comments on commit 5166414

Please sign in to comment.