need reduced echam and jsbach output for spinup #384
Comments
|
Can someone PLEASE PLEASE put this in? It's been on my list for months. |
|
How would this be best solved, Christopher? echam:
scenario: PI-CTRL-SPINUPOr rather: echam:
namelist_variant: spinup # Or reduced, minimal, or somethingI think using a new "scenario" may be confusing to some people |
|
I dont know. Someone of the esm tools heads need to decide that. Another point I cannot decide is how to distinguish different |
|
There is one problem to define scenario-dependent output streams of echam/jsbach. The streams:
- accw
- co2
- echam
- g3bid
- g3bim
- g3bday
- g3b1hi
- glday
- aclcim
- sp6h
- spim
streamsnc:
- aclcim
- g3b1hi
- g3bday
- g3bid
- g3bim
- glday
- glim
- jsbid
- sp6h
- spimdefine which output files will be moved from In my view the solution to this would be to consider all @mandresm @dbarbi could you please comment on that? Thanks a lot! |
|
Distinguishing that cleanly isn't really so easy. The Namelist for echam also influences jsbach output, but then we have two different yaml files and also two different namelists even just controlling model behaviour. Confusion lurks around every single corner there. Ultimately, we need to find some way of getting a single source of truth for the streams, and define that only once and have it propagate everywhere. Chris, did you make a branch? Given that we have space issues, I'd like to put that in and update the handbook |
|
Thanks @chrisdane for this suggestion, I think it's a very good point. About the streams, I would need to have a look again at the echam files. However, even if |
|
"Scenario" controls a lot more than just output, also for example which model features should be active. PI shouldn't have human land use change, a future run should. I would therefore recommend separating output control from the scenario. |
|
Thanks Miguel. I agree that having another @pgierz1: Having those @pgierz2: Yes I made a esm_tools branch and wrote that several times in my initial post: echam_scenario_PI-CTRL-SPINUP. @pgierz3: Separating output control and scenario is a good idea. But for echam/jsbach, the output control is defined in the namelist.echam. This namelist, in turn, enters the esm tools via |
|
Chris is correct, that would need a bit of a stronger rewrite. We could, for example, add one more layer of folders? Or maybe better, merge in the output part of the name list, separate from whatever defines physical behaviour? Do we have a merge feature for namelists yet? They are just dictionaries on the python side. I would again strongly suggest against making the output dependent on the scenarios. That (gut feeling) seems too easy to mess up accidentally. We will fix one, but forget the other. |
|
Chris, sorry I didn't see your branch in the initial post. Info is obviously there, I just need to learn how to read... |
|
Paul, the output definitions for echam/jsbach are not as simple as for fesom. Separating the stream definitions from everything else in the namelist.echam would yield more confusion in my view. For example, I dont see a problem in making output dependent on the scenario. In fact, my initial motivation is to have it. |
|
I think this discussion is very interesting and valuable, but I also think we need to include the main advance users of ECHAM into it if we are going to make a major change that affect the streams. What do you think? |
|
Sure please do so =) |
|
The original idea was to use the streams and streamsnc arrays to automatically GENERATE the output stream part in the namelist.echam, but we never got there. |
|
Ok. I dont think this is feasible. How would you specify that e.g. 1) only certain variables from a specfic stream should be saved 2) in a specific temporal interval? If the yaml lists |
|
initially, because you could use the same style of writing that to set the output of fesom, openifs, nemo, etc. without having to do it differently for each model. |
|
There is related problem with the historical echam namelist, i.e. &runctl
PUTDATA = 3,'hours','last',0
default_output = .true.and &mvstreamctl
interval = 1, 'days', 'last', 0
target = 'g3bid'
source = 'g3b'
variables = ... , 'temp2:mean', ...
/yield, for annual runs, the variable The annual ( The monthly ( The daily ( Since temp2 from the |
|
I can briefly comment on this one: If you are after monthly means, I have a few template namelists that we are ironing out here. Check out the "production" version, that might be what you need. You actually even made the spinup ones :-) https://gitlab.awi.de/paleodyn/Models/namelists Chris, to understand your screenshots (maybe I just need to read): those are anomalies between two files of the same run? We saw similar patterns comparing |
Yes. Same variable, same run, different output files. |
|
Alright, wrong place to complain, but: "ugh....echam....why" Consider the following more to be "public note taking" (or whatever thinking out loud is for a forum): For your screenshots, what you have is -- if I understand it correctly -- a yearly (or monthly, or daily) average of snapshots vs a yearly average of daily means. That would (maybe) explain the vertical bands you see there (day/night difference??) It's quite a small difference though, I would have expected something clearer if you're always capturing European midnight/3am/6am/noon. Long story short, depending on the analysis you want to do, I would instinctively prefer the data in the Yet another point on my ever-growing list of why we need to fix the echam namelist for sensible output.... |
For someone working with daily data the difference is on the order of 1 K, i.e. super large :p |
|
Yes, I was more referring to the top two figures. Funny how it is so localized over North America. I would have guessed that if you have 3 hour output and average noon/3pm/6pm/9pm/midnight/3am/6am/9am that there isn't such a clear spatial pattern. Plus you can see waves over Eurasia. Odd....but, as I said, I'd just use the data in the other file, that will be at least clearer. And still, one for the list: we need to tame echam output. Urgently. |
|
The default stream definitions are printed in every atmout. The whereas the that means In the echam6 docu its written that (p. 119)
And indeed, the default of
That would mean that for any variable, that is not explicitly defined via , one must check the respective atmout-entry to figure out if it represents accumulated ( |
|
One note: the outputs seems to be recorded for any changed stream. Not sure if the default is also written to the log. If it is, that'd be great to know, please also forward that info to Christian. At some point, we need to ask Hamburg to clarify. Perhaps that point has been reached. At least on my end, I'm out of expertise. Sorry. |
|
@mandresm @denizural @dbarbi could you please implement some kind of switch so that switching namelist.echam and/or namelist.jsbach becomes possible? The current workflow via |
|
You could try to implement that yourself. I can take 20 minutes tomorrow and show you the relevant code that would need to be modified. Open source works best if anyone who wants a feature tries to build it. That also would be good to increase overall knowledge of the code base by more than just the core team. So, if you want to try, pull develop, and start a new branch from there, open a draft PR, and via comments and whatever, we will talk you through it :-) |
|
Dropping echam.yaml:streams would be a large modification of the esm tools. I have not enough knowledge and time for this =) |
|
There is a relevant PR that allows you to use the echam namelist to automatically define the streams. You must set it up in your run config. Please see: esm-tools/esm_runscripts#165 Closing for now, re-open if more discussion is needed. |
|
The PR does not work. The following workaround quasi-works:
and
The following does not work yet:
|
|
I would not say the PR does not work. I would say it does not work yet ;-) Can you send me the path to your experiment? I will have a look. |
|
Correction: the 2 problems affect jsbach files only. |
|
Catching up on this: Does the reduced output look realistic in release 6? |
|
Due to the totally different output strategies of the individual models, I think a general out of-the-box implementation in the esm_tools is difficult. For fesom it works quite well via fesoms output scheduler and a yaml file listing all wanted variables in the wanted output frequencies. For echam/jsbach its more complicated. I found a workflow that suits for me, it works using
For other models I dont know. @JanStreffing: yes of course the output is realistic. |
|
Hi all, I've been looking at this issue for a bit and also to the solution @pgierz offered, that I believe never got merged (esm-tools/esm_runscripts#165). Is that correct? I see the solutions given to, and used by @chrisdane as provisional solutions, as in my eyes there is a "major" issue as @chrisdane points out, and it's that there is duplication of information. Not only that, but if you add a stream on a namelist and then you forget to update the stream list you won't realize unless you are looking at the log files very much in detail or you are monitoring the output and making sure it doesn't get dump into the I think the streams for I think it would also be great to add the stream's control through the runscript (still providing a good Please, let me know what do you think about these two ideas (getting a version of what Paul did into release 6 and stream-namelist modification through the runscript). In turn, if you are okay with your workflow now and you don't want any changes, you can go ahead and close the issue. |
|
Hi The PR esm-tools/esm_runscripts#165 does not work. I think its close to impossible to build an algorithm that deals with all possible namelist.echam stream definitions. Its a black box. I think the better solution is to put a list of So I am ok with closing, but this issue remains a dead end for the modularity approach of the esm_tools :( |
|
My idea was not to build a full syntax for the streams, but instead let the user include their modifications to the namelist through the runscript with the same structure of the namelists, with sections and variables, the same way that was done for The other point is the PR esm-tools/esm_runscripts#165. That one is broken, but is fixable as the problem it is trying to fix is relatively simple (at least as I understand it, maybe I am loosing something important): read the namelist and extract the stream names. This one might be something we want to pursue in the future. Anyway, let's see if someone reopens this one or something similar in the future. |
|
I would not say close to impossible. After all, there are rules inside of echam for how it produces output and what those files are called. Rules which we could replicate. Yes, it is very echam specific, but we already have specific model things on the tools. See for example Oasis. Plus Echam being sketchy about its documentation should be given as a lesson to anyone who thinks about writing climate code. Our own project is also slowly working on improving that. To me this boils down at the end to a design question. Duplicate information is by definition error prone. You are bound to forget one of your multiple places. I'd like to keep this issue open, as a place for discussion if nothing else. |
pgierz
added
Hackathon
|
This issue has been inactive for the last 365 days. It will now be marked as stale and closed after 30 days of further inactivity. Please add a comment to reset this automatic closing of this issue or close it if solved. |
Hi
Is your feature request related to a problem? Please describe.
The default
namelist.echamfor aPI-CTRLexperiment, which is (almost always?) also used for spinups, creates a lot of data on a high temporal interval (< month) due to the namelist blocksFor a spinup, this is unnecessary and bad practice since nobody will ever need this data but the disks are full with it.
Describe the solution you'd like
As far as I understand the esm tools, I would like to have a
esm_tools/namelists/echam/<version>/PI-CTRL-SPINUP/namelist.echamwhich is the same asesm_tools/namelists/echam/<version>/PI-CTRL/namelist.echambut with only a few important variables on monthly output frequency, or similar.I tried to achieve this. I ran a default echam-only
PI-CTRLexperiment on ollie:The resulting echam output after 1 month is
and for jsbach
To test for reduced echam output, I ran the experiment
The resulting echam output after 1 month is
and for jsbach
The total size of
outdataof 1 month reduces fromto
This reduction is roughly 100 - 4.6 MB / 367 MB * 100 ~ 99% although many of the most important variables are included. Of course, one could argue if e.g. the 3d variable
q(specific humidity) needs to be included or if other variables should be included and so on ...I would be very happy if you could implement this in some way as a standard for echam since I am really sick of this huge unnecessary spinup output everywhere.
Thanks a lot for consideration,
Chris
ps: I couldnt get the jsbach stream
vegto work (vegmonin the new namelist.echam in theecham_scenario_PI-CTRL-SPINUPbranch).The text was updated successfully, but these errors were encountered: