IGV: pull up computation of jvm heap memory into OOD job script #77
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
diedpigs
reviewed
Sep 23, 2021
| @@ -28,10 +28,19 @@ cd "${HOME}" | |||
| # Start Integrative Genomics Viewer | |||
| # | |||
|
|
|||
| # Determine memory for JVM in IGV | |||
| default_jvm_other_mb=512 # default to 0.5 GB | |||
| total_available_mb=$(($SLURM_MEM_PER_CPU * $SLURM_CPUS_PER_NODE)) | |||
There was a problem hiding this comment.
I don't see these two variables set in a job.
There was a problem hiding this comment.
didn't know this only set in a sbatch job, which is the way we start IGV job.
So I see it now.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
IGV uses Java, default JVM heap memory is 2 GB. For IGV jobs, the memory allocation must be communicated to JVM. We also can't provide 100% of the allocation to the heap without risking job termination for running out of memory. We have to leave some free for the JVM itself. If the allocation is more than 5GB, 512 MB is reserved for JVM, rest goes to heap. If less than 5GB, 90% goes to the heap. These are assumed-good values and have not been tested against IGV, so 512 MB may not be large enough to be robust, so we may see future errors.
See here for a more complete explanation: https://docs.uabgrid.uab.edu/wiki/Java#Xmx
Currently the computation is done in
igv.shfollowing modification of the script. The process for modification is not automated and so is prone to regression. We received an issue from a user about this following the most recent update of IGV. The immediate issue has been resolved, but not the root cause.This pull request pulls the memory computation up into the OOD job creation script, and passes the appropriate JVM flag to
igv.sh, which should pass it to the JVM. This should make OOD IGV robust against updates to IGV.There is some follow-on work to be done:
igv.shscripts in each version of IGV available in lmod. It isn't clear which flag would "win" or if both would be passed to the JVM. If both are passed, it isn't clear how the JVM will handle it.