-
Notifications
You must be signed in to change notification settings - Fork 11
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add longer tests to the test_consistent file #51
Comments
Per discussion on #54:
|
Hey @elbeejay I had a thought about this and wanted to run it by you. With Github Actions, we can schedule a job to run at a specified time. Let's say weekly at 1am EST on Saturday, when loads would otherwise be low. The job times out at 6 hours. We could schedule a job to run a model run with some config known to produce a "good" channel network, and let the job run weekly. When the job finishes (we should target a 4-5 hour run time) we can use this actions call to post the resulting "eta" map to a slack channel, and upload the artifacts (images, logs, etc) to check them out if there are any problems. I'm pretty sure this is fine and in line with Actions terms of service, but I would double check before doing anything. Thoughts? |
That sounds pretty neat. I guess the ultimate goal would be to tie in a set of tests and accomplish the longer run validation this way? With the aim of eventually moving away from a manual weekly check of the results and a more automated approach (maybe only running the long run if there was a push that week or something)? It seems like a good find and would give us a way to do these longer tests. The Actions documentation and terms of use seem pretty open about the types of workflows etc you run so long as you're under the usage limits. so I too am thinking this would be okay. |
Yeah, I wasn't really even thinking about explicit tests, but just thought it would be better than nothing to get a notification once a week with an image of a delta run from the latest build on That said, I guess there's really no reason we can't run a 4 hour job each time someone opens a PR, and somehow automate posting a comment to the PR with the delta... I checked on the ToS, and I think this use would be fine, because it is part of our "testing". Anyway, just some thoughts, I'd like to get to this eventually but it's probably not high priority. |
Longer tests will help make sure any cases are covered by consistency checks.
Use a few different model configurations and random seeds in different checks to increase redundancy in the consistency checking.
Original suggestion here
The text was updated successfully, but these errors were encountered: