Logging Validation loss in fine-grained level in Tensorboard logger.

[akashtadwai]

I wanted to log both the train and validation loss and other metrics similar to training loss (which is the default). I have looked into Strategy Logger but not very sure how to log both validation and training info. I ideally wanted some graph-like the following,

Can anyone guide me regarding what to do or how to achieve this?

[AndreaCossu]

Hi @akashtadwai ! The quickest way to produce a combined plot like that requires to use W&B Logger. The logger produces separate plots but the web interface allows you to combine them very easily.

Alternatively, you can create a custom logger which internally stores the metric values based on the name. E.g. Epoch_loss/train_phase/train_stream can be wait for Epoch_loss/eval_phase/test_stream and then you can log them together. However, loggers are built with the idea to immediately log things as soon as they arrive.

For ad-hoc plots, the best way is to logs values in a file (you can use TextLogger or build your own) and then parse it and create plots.

Let me know if you need further help on this 😄

[akashtadwai]

@AndreaCossu Actually, I was looking for something specific on how to log Loss_Epoch/test_phase/test_stream data on various levels. I think the default option for this is Loss_Epoch/train_phase/train_stream. How to mention specifically in W&B or Tensorboard to log both train and test streams acc/loss in all epochs.

[AndreaCossu]

If you want to perform evaluation during training and log also the evaluation metrics you just need to specify in the EvaluationPlugin instance:

1. Pass to the EvaluationPlugin constructor: loss_metrics(epoch=True, experience=True)
2. Pass to the CL strategy constructor: periodic_eval=1
3. Call strategy.train(exp, eval_streams=[scenario.test_stream])

In this way after each epoch you trigger evaluation and the logging of test metrics on all the test stream. Let me know if this solves your problem 😄

[akashtadwai]

2. periodic_eval=1

@AndreaCossu Isn't it eval_every? I see that _periodic_eval is a private ( strategies/base_strategy.py ) variable isn't it?

[AndreaCossu]

You are absolutely right, I meant eval_every, sorry about that 😄

[akashtadwai]

3. strategy.train(exp, eval_streams=[scenario.test_stream])

Hi, @AndreaCossu I have followed the exact steps but the validation logs were not found in the tensorboard logger. Here is a quick link of the tensorboard output (I have just trained it for 1 epoch for testing whether both train and validation are getting logged). I have written it as follows,

eval_plugin = EvaluationPlugin(
    accuracy_metrics(minibatch=True, epoch=True, experience=True, stream=True),
    loss_metrics(minibatch=True, epoch=True, experience=True, stream=True),
    timing_metrics(epoch=True, epoch_running=True),
    ExperienceForgetting(),
    cpu_usage_metrics(experience=True),
    StreamConfusionMatrix(num_classes=2, save_image=False),
    disk_usage_metrics(minibatch=True, epoch=True, experience=True, stream=True),
    loggers=[interactive_logger, text_logger, tb_logger]
)
cl_strategy = GEM(
    model, optimizer = Adam(model.parameters()),patterns_per_exp=4400,
    criterion = CrossEntropyLoss(), train_mb_size=128, train_epochs=1, eval_mb_size=128,eval_every=1,
    evaluator=eval_plugin,device=device)

res = cl_strategy.train(experience,eval_streams=[generic_scenario.test_stream])

[AndreaCossu]

I see. You are evaluating on the test stream so the values are put in the related curve of eval_phase/test_stream. You should see more points there than you'd expect.
You can call the periodic eval on a separate validation stream and that should appear separated from the rest of the curves.
Currently, the periodic eval within training is just a regular eval.

@lrzpellegrini can you provide him a short way to build the validation stream?
@AntonioCarta I think this may cause future misunderstanding. Should we better document it inside the training tutorial? I can then point to it in the evaluation and logging tutorial.

[AndreaCossu]

Maybe instead of adding it to the doc we can write an example out of this: how to exploit periodic eval + custom streams to monitor validation curves. I think it is a common feature people would like to grasp quickly.

[akashtadwai]

You can call the periodic eval on a separate validation stream and that should appear separated from the rest of the curves.
Currently, the periodic eval within the training is just a regular eval.

Hi, @AndreaCossu if possible can you provide a brief description of this on how to perform validation loss logging 😅

[AndreaCossu]

If you set eval_every=1 and train with

for i, exp in enumerate(scenario.train_stream):
        cl_strategy.train(exp, eval_streams=[scenario.test_stream[i]])

you can retrieve the metrics on the test stream at the end of each epoch by using this CSV logger (currently set for loss and accuracy only). Let me know if this works for you 😄

import sys
from typing import List, TYPE_CHECKING

import torch
import os

from avalanche.evaluation.metric_results import MetricValue
from avalanche.logging import StrategyLogger

if TYPE_CHECKING:
    from avalanche.training import BaseStrategy


class CSVLogger(StrategyLogger):
    """
    The `CSVLogger` logs accuracy and loss metrics into a csv file.
    Metrics are logged separately for training and evaluation in files
    training_results.csv and eval_results.csv, respectively.

    The user should monitor at least Epoch and Experience accuracy and loss.
    Training should also monitor the performance on  test experience associated
    to the current training experience (i.e. eval_streams=[scenario.test_stream[i]]).
    If not provided, validation loss and accuracy will be logged as zero.

    The training file header is composed of:
    training_exp_id, epoch, training_accuracy, val_accuracy, training_loss, val_loss.
    The evaluation file header is composed of:
    eval_exp, training_exp, eval_accuracy, eval_loss, forgetting
    """

    def __init__(self, log_folder=None):
        """
        Creates an instance of `CSVLogger` class.

        :param log_folder: folder in which to create log files.
            If None, `csvlogs` folder in the default current directory
            will be used.
        """

        super().__init__()
        self.log_folder = log_folder if log_folder is not None else "csvlogs"
        self.training_file = open(os.path.join(self.log_folder, 'training_results.csv'), 'w')
        self.eval_file = open(os.path.join(self.log_folder, 'eval_results.csv'), 'w')
        os.makedirs(self.log_folder, exist_ok=True)
        self.training_exp_id = None  # current training experience id
        self.val_acc, self.val_loss = 0, 0  # validation metrics computed during training

        # print csv headers
        print('training_exp', 'epoch', 'training_accuracy', 'val_accuracy',
              'training_loss', 'val_loss', sep=',', file=self.training_file, flush=True)
        print('eval_exp', 'training_exp', 'eval_accuracy', 'eval_loss', 'forgetting',
              sep=',', file=self.eval_file, flush=True)

    def log_metric(self, metric_value: 'MetricValue', callback: str) -> None:
        pass

    def _val_to_str(self, m_val):
        if isinstance(m_val, torch.Tensor):
            return '\n' + str(m_val)
        elif isinstance(m_val, float):
            return f'{m_val:.4f}'
        else:
            return str(m_val)

    def print_train_metrics(self, training_exp, epoch, train_acc, val_acc, train_loss, val_loss):
        print(training_exp, epoch, self._val_to_str(train_acc),
              self._val_to_str(val_acc), self._val_to_str(train_loss),
              self._val_to_str(val_loss), sep=',', file=self.training_file, flush=True)

    def print_eval_metrics(self, eval_exp, training_exp, eval_acc, eval_loss, forgetting):
        print(eval_exp, training_exp, self._val_to_str(eval_acc),
              self._val_to_str(eval_loss), self._val_to_str(forgetting),
              sep=',', file=self.eval_file, flush=True)

    def after_training_epoch(self, strategy: 'BaseStrategy',
                             metric_values: List['MetricValue'], **kwargs):
        super().after_training_epoch(strategy, metric_values, **kwargs)
        train_acc, val_acc, train_loss, val_loss = 0, 0, 0, 0
        for val in metric_values:
            if 'train_stream' in val.name:
                if val.name.startswith('Top1_Acc_Epoch'):
                    train_acc = val.value
                elif val.name.startswith('Loss_Epoch'):
                    train_loss = val.value

        self.print_train_metrics(self.training_exp_id, strategy.epoch,
                                 train_acc, self.val_acc, train_loss, self.val_loss)
        self.val_acc, self.val_loss = 0, 0

    def after_eval_exp(self, strategy: 'BaseStrategy',
                       metric_values: List['MetricValue'], **kwargs):
        super().after_eval_exp(strategy, metric_values, **kwargs)
        acc, loss, forgetting = 0, 0, 0

        for val in metric_values:
            if not strategy.is_eval:  # validation within training
                if val.name.startswith('Top1_Acc_Exp'):
                    self.val_acc = val.value
                elif val.name.startswith('Loss_Exp'):
                    self.val_loss = val.value
            else:
                if val.name.startswith('Top1_Acc_Exp'):
                    acc = val.value
                elif val.name.startswith('Loss_Exp'):
                    loss = val.value
                elif val.name.startswith('ExperienceForgetting'):
                    forgetting = val.value

        if strategy.is_eval:
            self.print_eval_metrics(strategy.experience.current_experience,
                                    self.training_exp_id, acc, loss, forgetting)

    def before_training_exp(self, strategy: 'BaseStrategy',
                            metric_values: List['MetricValue'], **kwargs):
        super().before_training(strategy, metric_values, **kwargs)
        self.training_exp_id = strategy.experience.current_experience

    def close(self):
        self.training_file.close()
        self.eval_file.close()

[vlomonaco]

@akashtadwai does the answer from @AndreaCossu answer your information need? I'll mark this question as answered feel free to re-open it if you need more info!

[AndreaCossu]

By the way, the latest PR adds the CSVLogger in the logging module so you can use it out of the box 😄