Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

C48mx500_hybAOWCDA full-cycle enkfgdas forecasts fail #1429

Open
RussTreadon-NOAA opened this issue Dec 29, 2024 · 5 comments · May be fixed by #1430
Open

C48mx500_hybAOWCDA full-cycle enkfgdas forecasts fail #1429

RussTreadon-NOAA opened this issue Dec 29, 2024 · 5 comments · May be fixed by #1430

Comments

@RussTreadon-NOAA
Copy link
Contributor

When g-w CI C48mx500_hybAOWCDA is run using GDASApp at eec2e1b, the 20210325 00Z enkfgdas_fcst_mem001 and enkfgdas_fcst_mem002 abort with the message

14:
14: FATAL from PE     2: initialize_oda_incupd_file: Array size mismatch for oda data.
14:
14: Image              PC                Routine            Line        Source
14: gfs_model.x        0000000005651AA9  mpp_mod_mp_mpp_er          72  mpp_util_mpi.inc
14: gfs_model.x        0000000004A2ADE0  mom_error_handler         191  MOM_error_handler.F90
14: gfs_model.x        0000000004B72B0C  mom_state_initial        2285  MOM_state_initialization.F90
14: gfs_model.x        0000000004B89FF1  mom_state_initial         669  MOM_state_initialization.F90
14: gfs_model.x        000000000475BBE5  mom_mp_initialize        2999  MOM.F90
14: gfs_model.x        00000000047086A4  mom_ocean_model_n         282  mom_ocean_model_nuopc.F90
14: gfs_model.x        00000000046DCD16  mom_cap_mod_mp_in         664  mom_cap.F90
14: gfs_model.x        0000000000ADB3B4  Unknown               Unknown  Unknown
14: gfs_model.x        0000000000ADF48F  Unknown               Unknown  Unknown
14: gfs_model.x        000000000114F9AA  Unknown               Unknown  Unknown
14: gfs_model.x        00000000012E45C8  Unknown               Unknown  Unknown
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.

If C48mx500_hybAOWCDA is rerun using GDASApp at d6097af, the C48mx500_hybAOWCDA case successfully runs to completion.

Is it possible that GDASApp PR #1417 requires additional changes to GDASApp, submodules, and/or g-w to create a MOM6 increment which the forecast model can use?
@RussTreadon-NOAA RussTreadon-NOAA mentioned this issue Dec 29, 2024
Open
@RussTreadon-NOAA
Copy link
Contributor Author

Attention: @guillaumevernieres , @AndrewEichmann-NOAA

@shlyaeva
Copy link
Collaborator

I expect that the 3DVar case (C48mx500_3DVarAOWCDA) runs correctly? I wonder if not everything got updated for the ensembles in #1417 and companions. Guillaume is on PTO this week, I'm also happy to help look into it on Monday.

@RussTreadon-NOAA
Copy link
Contributor Author

Yes, C48mx500_3DVarAOWCDA successfully runs to completion

orion-login-3:/work2/noaa/stmp/rtreadon/EXPDIR/C48mx500_3DVarAOWCDA_test$ rocotostat -d C48mx500_3DVarAOWCDA_test.db -w C48mx500_3DVarAOWCDA_test.xml -c all -s
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103241800        Done    Dec 28 2024 13:12:33    Dec 28 2024 13:30:08
202103250000        Done    Dec 28 2024 13:12:33    Dec 28 2024 14:45:06

It seems something for the ensembles is missing in #1417 or g-w

@shlyaeva
Copy link
Collaborator

I think this error happens because we don't have ssh in the mom6 increments for ensembles; I'll keep looking why.

@RussTreadon-NOAA
Copy link
Contributor Author

Thank you, @shlyaeva, for the update and troubleshooting this issue.

@shlyaeva shlyaeva linked a pull request Dec 31, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Bugfix for ensemble update: rollback changes in gdas_ens_handler NOAA-EMC/GDASApp
2 participants
@shlyaeva @RussTreadon-NOAA