Injection/multispecies testing for others

README.md

@@ -73,3 +73,44 @@ Most subdirectories contain a run.sh script that outlines how to run the problem
 
 The case can the be run similar to other MIRGE-Com applications.
 For examples see the MIRGE-Com [documentation](https://mirgecom.readthedocs.io/en/latest/running/systems.html)
+
+Notes for running with multispecies:
+
+Some multispecies tests are in the following directories:
+```
+smoke_test_injection_2d
+smoke_test_injection_3d
+```
+
+The 3D multispecies tests are currently the best approximation to the expected Y2 prediction workload.  The
+general procedure for preparing and running the tests in these directories is as follows:
+1. Generate the mesh
+```
+cd data&&./make_mesh.sh
+```
+2. Choose inert or with combustion:
+```
+ln -sf run_params_combustion.yaml run_params.yaml
+-or-
+ln -sf run_params_inert.yaml run_params.yaml
+```
+3. Initialize the case solution (make sure to run this on the same number of ranks as you plan to run):
+```
+run_init.sh
+```
+4. Run the simulation by restarting from the init'd soln:
+```
+run.sh
+```
+
+For production scale testing see the test_mesh_injection directory. 
+
+```test_mesh_injection/template```
+
+There are 4 run sizes here, each with their own meshing. These meshes can be gererated similar to above. On Lassen the user can find pre-generated meshes in 
+
+```/usr/WS1/xpacc/CEESD/Y2_test_meshes```
+
+The meshes range in size from 1 million elements (eigthX) to 180 millions elements (oneX). All runs are set to use multi-species with chemistry (nspecies=7).
+
+A sample Lassen submission script is co-located with the runs, combinedLassenBatch.sh.

smoke_test_injection_3d/lassen.bsub.sh

@@ -0,0 +1,51 @@
+#!/bin/bash
+#BSUB -nnodes 1                   # number of nodes
+#BSUB -W 120                       # walltime in minutes
+#BSUB -q pbatch                   # queue to use
+
+
+set -ex
+
+# Run this script with 'bsub lassen.bsub.sh'
+
+# Put any environment activation here, e.g.:
+# source ../../config/activate_env.sh
+
+# OpenCL device selection:
+export PYOPENCL_CTX="port:tesla"      # Run on Nvidia GPU with pocl
+# export PYOPENCL_CTX="port:pthread"  # Run on CPU with pocl
+
+nnodes=$(echo $LSB_MCPU_HOSTS | wc -w)
+nnodes=$((nnodes/2-1))
+nproc=$((4*nnodes)) # 4 ranks per node, 1 per GPU
+
+echo nnodes=$nnodes nproc=$nproc
+
+# -a 1: 1 task per resource set
+# -g 1: 1 GPU per resource set
+# -n $nproc: $nproc resource sets
+jsrun_cmd="jsrun -g 1 -a 1 -n $nproc"
+
+# See
+# https://mirgecom.readthedocs.io/en/latest/running.html#avoiding-overheads-due-to-caching-of-kernels
+# on why this is important
+export XDG_CACHE_HOME="/tmp/$USER/xdg-scratch"
+
+# Fixes https://github.com/illinois-ceesd/mirgecom/issues/292
+# (each rank needs its own POCL cache dir)
+export POCL_CACHE_DIR_ROOT="/tmp/$USER/pocl-cache"
+
+# Print task allocation
+$jsrun_cmd js_task_info
+
+echo "----------------------------"
+
+# Run application
+# -O: switch on optimizations
+# POCL_CACHE_DIR=...: each rank needs its own POCL cache dir
+
+# isolator_injection_init.py must be run with the same config as isolator_injection_run.py, in particular:
+# - same numbers of ranks in both cases
+# - lazy runs need a lazy init
+$jsrun_cmd bash -c 'POCL_CACHE_DIR=$POCL_CACHE_DIR_ROOT/$$ python -m mpi4py isolator_injection_init.py -i run_params.yaml --lazy'
+$jsrun_cmd bash -c 'POCL_CACHE_DIR=$POCL_CACHE_DIR_ROOT/$$ python -m mpi4py isolator_injection_run.py -r restart_data/isolator_init-000000 -i run_params.yaml --lazy'

smoke_test_injection_3d/run_params_combustion.yaml

@@ -0,0 +1,22 @@
+nviz: 10
+nrestart: 10
+nhealth: 1
+nstatus: 1
+current_dt: 1.0e-9
+t_final: 2.e-8
+order:  1
+dimen: 3
+mach_inj: 0.8
+nspecies: 7
+#nspecies: 2
+#nspecies: 0
+alpha_sc: 0.1
+s0_sc: -5.0
+kappa_sc: 0.5
+integrator: euler
+#health_pres_min: 1
+#health_pres_max: 500000
+health_pres_min: 2.65231e3
+health_pres_max: 2.77231e5
+health_mass_frac_min: -9.e-7
+health_mass_frac_max: 1.0001