Issue on checkpoints

[XYJ-NB]

Thanks for sharing this project. It is great to see such a powerful foundation model!

I have been testing the open-source model and the corresponding checkpoints provided by your team in ERA5. We used the ERA5 data of May 2024 as input and followed the steps in https://microsoft.github.io/aurora/example_era5.html. We found that RMSE was similar to IFS, but there was a gap between GraphCast. The results from our experiments have not been as promising, and we are unsure if this might be due to the checkpoint we are using or if there are specific aspects of the model or experimental setup that we may have overlooked.

Could you kindly confirm whether the checkpoint we are using corresponds to the model that delivered the best results in your published work? If not, are there other checkpoints or configurations that we should consider to replicate the performance presented in the paper? Additionally, any insights on potential pitfalls or nuances in the implementation would be greatly appreciated.

[wesselb]

Hey @XYJ-NB! Thanks for opening an issue. :)

The checkpoint that you've been using is the generic pretrained checkpoint. This checkpoint has not been specifically fine-tuned for ERA5, so performance is expected to be OK, but not optimal.

Our best models are the fine-tuned models. In your case, I think that the 0.25-degree model fine-tuned to HRES T0 is what you're looking for. This model should outperform GraphCast in terms of RMSE.

[XYJ-NB]

Hey @XYJ-NB! Thanks for opening an issue. :)

The checkpoint that you've been using is the generic pretrained checkpoint. This checkpoint has not been specifically fine-tuned for ERA5, so performance is expected to be OK, but not optimal.

Our best models are the fine-tuned models. In your case, I think that the 0.25-degree model fine-tuned to HRES T0 is what you're looking for. This model should outperform GraphCast in terms of RMSE.

Thank you for your reply, but I noticed that the fine-tuned models are for HRES T0, so don't we need to consider the difference in ERA5 reanalysis data?

[wesselb]

@XYJ-NB You're right that this would run the model on another source of data, namely HRES T0 instead of ERA5. The main reason why we did not fine-tune to ERA5 is that ERA5 is a reanalysis product (instead of an analysis product like HRES T0), meaning that it is not available operationally. In addition, for years 2016 and later, HRES T0 is considered to be more accurate than ERA5 because HRES is run at 0.1 degrees instead of 0.25 degrees.