EarlyStoppingPlugin: incompatible callbacks for template?

[highclef]

Hello,

I tried to apply EarlyStoppingPlugin to one of the CL strategy, but it seems to be not working.
It does not stop after the patience, but just keeps going.
This is my code and the output message at the beginning.
Is EarlyStoppingPlugin currently not working for the strategy templates? How do I fix this?

eval_plugin_si = EvaluationPlugin(
    loss_metrics(minibatch=True, epoch=True, experience=True, stream=True),
    forgetting_metrics(experience=True, stream=True),
    loggers=loggers_si,
    strict_checks=None
)

strategy = SynapticIntelligence(
     model, optimizer, criterion, train_mb_size=32, train_epochs=70,
     eval_mb_size=32, si_lambda=54.76850169265467, evaluator=eval_plugin_si, 
     plugins=[EarlyStoppingPlugin(patience=10, val_stream_name='train')])

results = []

for train_task in train_stream:
     strategy.train(train_task, num_workers=4)
     results.append(strategy.eval(test_stream))

/usr/local/lib/python3.7/dist-packages/avalanche/training/templates/base.py:206: UserWarning: Plugin <avalanche.training.plugins.early_stopping.EarlyStoppingPlugin object at 0x7f652322b9d0> implements incompatible callbacks for template <avalanche.training.supervised.strategy_wrappers.SynapticIntelligence object at 0x7f652322b850>. This may result in errors.
  f"Plugin {p} implements incompatible callbacks for template"
/usr/local/lib/python3.7/dist-packages/avalanche/training/plugins/early_stopping.py:95: UserWarning: Metric Top1_Acc_Stream used by the EarlyStopping plugin is not computed yet. EarlyStopping will not be triggered.
  f"Metric {self.metric_name} used by the EarlyStopping plugin "

[AntonioCarta]

By default EarlyStopping is looking for the Top1_Acc_Stream metric. However, you are not computing it, as you can see from the second warning.

You should use

eval_plugin_si = EvaluationPlugin(
    accuracy_metrics(stream=True, ...),   # add accuracy metrics
    loss_metrics(minibatch=True, epoch=True, experience=True, stream=True),
    forgetting_metrics(experience=True, stream=True),
    loggers=loggers_si,
    strict_checks=None
)

[evertonaleixo]

Another option is to choose another metric for EarlyStoppingPluging monitor, using the parameter 'metric_name' in the constructor.