Issue on page /example_era5.html

[mr-arashmousavi]

To predict weather data we need to provide some historical data in the format which is mentioned in this link:
Example: Predictions for ERA5 — Aurora: A Foundation Model of the Atmosphere
But one item that I need to set in meta data apart from lat and long is time which I believe is forecast time.
metadata=Metadata(
lat=torch.from_numpy(surf_vars_ds.latitude.values),
lon=torch.from_numpy(surf_vars_ds.longitude.values),
time=(surf_vars_ds.valid_time.values.astype("datetime64[s]").tolist()[i],),
atmos_levels=tuple(int(level) for level in atmos_vars_ds.pressure_level.values),
),
but why would I want to forecast weather in the past, I already know what the weather was in the past and in fact I am providing that to model. I expect the model to be able to forecast weather data in future let say at lest say 1 hour from now. but when I pass any forecast date in future, or even in last few days it says it can forecast data 7 days ago or something. What am I missing?

[wesselb]

Hey @mr-arashmousavi! Thanks for opening an issue. :)

Metadata.time denotes the time of the data that you give to the model. The model needs to know what time corresponds to its inputs, because e.g. it matters whether it is day or night.

but why would I want to forecast weather in the past, I already know what the weather was in the past and in fact I am providing that to model.

Aurora is able to predict the future state of the weather by taking in a state in the past and "evolving" that state forward in time. More precisely, for every application of the model, Aurora advances the weather state by 6 hours.

For example, if you give Aurora the weather at 12:00 on 17 Dec (yesterday) and you apply the model eight times, then this means that the model produces predictions for 8 * 6 hours = 48 hours in the future, so the resulting predictions are for 12:00 on 19 Dec (tomorrow).

You should use aurora.rollout to apply the model successively to produce long-term predictions.

[mr-arashmousavi]

Thanks for your prompt response, in applied your suggestion and I extended the forecast horizon (Steps in rollout function) and that forecasted multiple days.
now the problem is the data that I can fetch using that source is provided in the example has 5 or 6 days lag
for ex: that is the max date I could pas

c.retrieve(
    "reanalysis-era5-single-levels",
    {
        "product_type": "reanalysis",
        "variable": [
            "geopotential",
            "land_sea_mask",
            "soil_type",
        ],
        "year": "2024",
        "month": "12",
        "day": "13",
        "time": "00:00",
        "format": "netcdf",
    },
    str(download_path / "static2.nc"),
)

so even if predict next 5 days still is not current. if I ask for 10 days prediction, sure it would include future from now(dec 13 + 10 days), but I think that stale data would impact the forecast performance. I am assuming as we go more into future the quality of forecast is worsening

so in brief the question is how can I get more recent data? this way the forecast would be from now to 1 day in future which is what I need.
Thanks a lot in advance

[wesselb]

@mr-arashmousavi, unfortunately ERA5 necessarily has a delay. There is no way of getting ERA5 in real time.

To produce better forecasts based on more recent data, it it recommended to use the model fine-tuned for HRES T0. Please see the relevant notebook. HRES T0 is also available in nearly real time (and also higher quality than ERA5).

The issue with HRES T0 is that it is not available publicly. To get HRES T0, you will need an ECMWF licence. See this page.