Speed up GSI analysis jobs in CI testing #3115

CoryMartin-NOAA · 2024-11-20T18:51:30Z

Description

This PR adds DO_TEST_MODE, which can be used for other things in the future but for now sets the GSI to run just 5 iterations per outer loop to reduce runtime for CI testing.

Resolves #3114

Type of change

Bug fix (fixes something broken)
New feature (adds functionality)
Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? MAYBE
Does this change require an update to any of the following submodules? NO

How has this been tested?

Clone and build on Hera
Cycled test on Hera

Checklist

RussTreadon-NOAA

Straightforward PR. Approve ... but did not test.

ci/cases/yamls/atmaerosnowDA_defaults_ci.yaml

emcbot · 2024-11-22T15:20:41Z

Experiment C48_S2SW FAILED on Hera in Build# 2 with error logs:

/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C48_S2SW_0bb9078e/logs/2021032312/gfs_stage_ic.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2024-11-22T15:20:47Z

Experiment C48_S2SW FAILED on Hera in Build# 2 in
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/EXPDIR/C48_S2SW_0bb9078e

JessicaMeixner-NOAA · 2024-11-22T15:33:14Z

@WalterKolczynski-NOAA I was pointed to this failure offline because it was about a wave IC issue:

SError: unable to copy /scratch1/NCEPDEV/global/glopara/data/ICSDIR/C48mx500/20240610/gfs.20210323/06/model/wave/restart/20210323.120000.restart.ww3 to /scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C48_S2SW_0bb9078e/gfs.20210323/06//model/wave/restart/20210323.120000.restart.ww3

I'm concerned that changes in staging of wave ICs has messed up existing regression tests in prep for #3112
removing the grid id perhaps is causing an issue because we have many wave gridids that are used in various places and what was copied to ww3? And where? I'll continue this conversation in 3112. I don't think we want to break other tests before 3112 is merged do we? And perahps we need to carefully go through and make sure ICs are properly placed for various wave grids for various CI tests. I'm unsure about this now.

emcbot · 2024-11-22T15:45:22Z

Experiment C48_S2SWA_gefs FAILED on Hera in Build# 2 with error logs:

/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C48_S2SWA_gefs_0bb9078e/logs/2021032312/gefs_stage_ic.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2024-11-22T15:45:28Z

Experiment C48_S2SWA_gefs FAILED on Hera in Build# 2 in
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/EXPDIR/C48_S2SWA_gefs_0bb9078e

WalterKolczynski-NOAA · 2024-11-22T17:05:49Z

@WalterKolczynski-NOAA I was pointed to this failure offline because it was about a wave IC issue:

SError: unable to copy /scratch1/NCEPDEV/global/glopara/data/ICSDIR/C48mx500/20240610/gfs.20210323/06/model/wave/restart/20210323.120000.restart.ww3 to /scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C48_S2SW_0bb9078e/gfs.20210323/06//model/wave/restart/20210323.120000.restart.ww3

I'm concerned that changes in staging of wave ICs has messed up existing regression tests in prep for #3112 removing the grid id perhaps is causing an issue because we have many wave gridids that are used in various places and what was copied to ww3? And where? I'll continue this conversation in 3112. I don't think we want to break other tests before 3112 is merged do we? And perahps we need to carefully go through and make sure ICs are properly placed for various wave grids for various CI tests. I'm unsure about this now.

A few symlinks got broken somehow. Breaking old tests was not intentional. The links are restored now.

emcbot · 2024-11-22T20:28:07Z

Experiment C96C48_ufs_hybatmDA FAILED on Hera in Build# 3 in
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/EXPDIR/C96C48_ufs_hybatmDA_0bb9078e

emcbot · 2024-11-22T20:28:07Z

Experiment C96C48_hybatmDA FAILED on Hera in Build# 3 in
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/EXPDIR/C96C48_hybatmDA_0bb9078e

emcbot · 2024-11-22T20:28:18Z

CI Failed on Hera in Build# 3
Built and ran in directory /scratch1/NCEPDEV/global/CI/3115


Experiment C96_S2SWA_gefs_replay_ics_0bb9078e Completed 1 Cycles: *SUCCESS* at Fri Nov 22 19:26:19 UTC 2024
Experiment C96_atm3DVar_0bb9078e Terminated with 0
FAIL
FAIL tasks failed and 20 dead at Fri Nov 22 19:56:33 UTC 2024
Experiment C96_atm3DVar_0bb9078e Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f000.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f003.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f006.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f009.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f012.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f015.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f018.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f021.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f024.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f027.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f030.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f033.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f036.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f039.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f042.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f045.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f048.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f051.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f054.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f057.log
Experiment C48mx500_3DVarAOWCDA_0bb9078e Completed 2 Cycles: *SUCCESS* at Fri Nov 22 20:02:37 UTC 2024
Experiment C48_ATM_0bb9078e Completed 2 Cycles: *SUCCESS* at Fri Nov 22 20:02:38 UTC 2024

emcbot · 2024-11-22T20:32:48Z

Checkout Failed on Hera in Build# 5: null

emcbot · 2024-11-22T20:32:49Z

Checkout Failed on Hera in Build# 5: null

emcbot · 2024-11-23T15:30:26Z

CI Passed on Hera in Build# 7
Built and ran in directory /scratch1/NCEPDEV/global/CI/3115


Experiment C96_S2SWA_gefs_replay_ics_baccdedd Completed 1 Cycles: *SUCCESS* at Sat Nov 23 13:15:45 UTC 2024
Experiment C48mx500_3DVarAOWCDA_baccdedd Completed 2 Cycles: *SUCCESS* at Sat Nov 23 13:27:55 UTC 2024
Experiment C48_ATM_baccdedd Completed 2 Cycles: *SUCCESS* at Sat Nov 23 13:33:59 UTC 2024
Experiment C96C48_hybatmaerosnowDA_baccdedd Completed 3 Cycles: *SUCCESS* at Sat Nov 23 14:47:18 UTC 2024
Experiment C96C48_hybatmDA_baccdedd Completed 3 Cycles: *SUCCESS* at Sat Nov 23 14:47:19 UTC 2024
Experiment C48_S2SWA_gefs_baccdedd Completed 1 Cycles: *SUCCESS* at Sat Nov 23 14:48:19 UTC 2024
Experiment C96_atm3DVar_baccdedd Completed 3 Cycles: *SUCCESS* at Sat Nov 23 14:53:11 UTC 2024
Experiment C48_S2SW_baccdedd Completed 2 Cycles: *SUCCESS* at Sat Nov 23 15:23:38 UTC 2024
Experiment C96C48_ufs_hybatmDA_baccdedd Completed 3 Cycles: *SUCCESS* at Sat Nov 23 15:30:11 UTC 2024

…est (#3120) This PR is a companion to NOAA-EMC/GDASApp#1365 It turns ```C96C48_ufs_hybatmDA``` and ```C48mx500_3DVarAOWCDA``` into a regression test using the JEDI application testing feature. This feature is turned on using the new ```DO_TEST_MODE``` parameter added to ```config.base``` in GW PR [#3115](#3115). This parameter is set to ```"YES"``` in the yaml defaults for the JEDI-based CI tests in GW. The motivation for this PR is a need to catch changes in JEDI which alter the outputs of our applications. --------- Co-authored-by: CoryMartin-NOAA <[email protected]> Co-authored-by: RussTreadon-NOAA <[email protected]>

CoryMartin-NOAA added 2 commits November 20, 2024 14:49

Add DO_TEST_MODE option and reduce niter

70af6af

update configs

0bb9078

CoryMartin-NOAA self-assigned this Nov 20, 2024

CoryMartin-NOAA requested review from TerrenceMcGuinness-NOAA and WalterKolczynski-NOAA as code owners November 20, 2024 18:51

CoryMartin-NOAA requested a review from RussTreadon-NOAA November 20, 2024 18:53

RussTreadon-NOAA approved these changes Nov 20, 2024

View reviewed changes

CoryMartin-NOAA requested a review from DavidNew-NOAA November 21, 2024 18:49

DavidNew-NOAA reviewed Nov 21, 2024

View reviewed changes

ci/cases/yamls/atmaerosnowDA_defaults_ci.yaml Show resolved Hide resolved

DavidNew-NOAA approved these changes Nov 22, 2024

View reviewed changes

DavidHuber-NOAA approved these changes Nov 22, 2024

View reviewed changes

DavidHuber-NOAA added the CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules label Nov 22, 2024

DavidHuber-NOAA added the CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera label Nov 22, 2024

emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Nov 22, 2024

JessicaMeixner-NOAA mentioned this pull request Nov 22, 2024

Fix wave restart for cold start and add ic version file #3112

Merged

11 tasks

emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Nov 22, 2024

WalterKolczynski-NOAA added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Nov 22, 2024

emcbot added CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera labels Nov 22, 2024

emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera labels Nov 22, 2024

WalterKolczynski-NOAA added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Nov 22, 2024

emcbot added CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera labels Nov 22, 2024

WalterKolczynski-NOAA added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera and removed CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera labels Nov 22, 2024

WalterKolczynski-NOAA approved these changes Nov 23, 2024

View reviewed changes

WalterKolczynski-NOAA merged commit efc25be into NOAA-EMC:develop Nov 23, 2024
10 of 11 checks passed

DavidNew-NOAA mentioned this pull request Nov 24, 2024

Turn C96C48_ufs_hybatmDA and C48mx500_3DVarAOWCDA into a regression test #3120

Merged

22 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Speed up GSI analysis jobs in CI testing #3115

Speed up GSI analysis jobs in CI testing #3115

CoryMartin-NOAA commented Nov 20, 2024 •

edited

Loading

RussTreadon-NOAA left a comment

emcbot commented Nov 22, 2024

emcbot commented Nov 22, 2024

JessicaMeixner-NOAA commented Nov 22, 2024

emcbot commented Nov 22, 2024

emcbot commented Nov 22, 2024

WalterKolczynski-NOAA commented Nov 22, 2024

emcbot commented Nov 22, 2024

emcbot commented Nov 22, 2024

emcbot commented Nov 22, 2024

emcbot commented Nov 22, 2024

emcbot commented Nov 22, 2024

emcbot commented Nov 23, 2024

Speed up GSI analysis jobs in CI testing #3115

Speed up GSI analysis jobs in CI testing #3115

Conversation

CoryMartin-NOAA commented Nov 20, 2024 • edited Loading

Description

Type of change

Change characteristics

How has this been tested?

Checklist

RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

emcbot commented Nov 22, 2024

emcbot commented Nov 22, 2024

JessicaMeixner-NOAA commented Nov 22, 2024

emcbot commented Nov 22, 2024

emcbot commented Nov 22, 2024

WalterKolczynski-NOAA commented Nov 22, 2024

emcbot commented Nov 22, 2024

emcbot commented Nov 22, 2024

emcbot commented Nov 22, 2024

emcbot commented Nov 22, 2024

emcbot commented Nov 22, 2024

emcbot commented Nov 23, 2024

CoryMartin-NOAA commented Nov 20, 2024 •

edited

Loading