codes dumpi workload

Inspecting DUMPI Traces

Individual operations in the DUMPI traces can be inspected by using the codes-workload-dump utility (in src/workload). This utility provides detailed information on each of the MPI operations in the logs e.g. operation type (whether send/recv/wait/collective), tag number, number of bytes transmitted (derived from the data type and count of the MPI operation), start and end times of operation.

Note that running this utility on a large number of traces will generate a lot of output so it is preferred to run it on a small number of MPI ranks (<4).

./bin/codes-workload-dump --type "dumpi-trace-workload" --num-ranks n --dumpi-log dumpi-trace-dir/dumpi-2014.03.06.20.34.26-

Replaying Application Traces on Network Models

Our primary network workload generator is via the DUMPI tool (http://sst.sandia.gov/about_dumpi.html). DUMPI collects and reads events from MPI applications. See the DUMPI documentation for how to generate traces. There are additionally publically-available traces at http://portal.nersc.gov/project/CAL/designforward.htm

Note on trace reading - the input file prefix to the dumpi workload generator should be everything up to the rank number. E.g., if the dumpi files are of the form "dumpi-YYYY.MM.DD.HH.MM.SS-XXXX.bin", then the input should be "dumpi-YYYY.MM.DD.HH.MM.SS-"

Replaying Application Trace on CODES MPI Simulation Layer

• Download and untar the DUMPI AMG application trace for 216 MPI ranks using the following download link:

wget http://portal.nersc.gov/project/CAL/doe-miniapps-mpi-traces/AMG/df_AMG_n216_dumpi.tar.gz

• Configure model-net config file (For this example config file is available at src/network-workloads/conf/modelnet-mpi-test-dfly-amg-216.conf)

• Run the DUMPI trace replay simulation on top of model-net using: (/dumpi-2014.03.03.14.55.23- is the prefix of the DUMPI trace file. We skip the last 4 digit prefix of the DUMPI trace files).

./install/bin/model-net-mpi-replay --sync=1 --num_net_traces=216 --workload_file=/path/to/dumpi/trace/directory/dumpi-2014.03.03.15.09.03-
--workload_type="dumpi" --lp-io-dir=amg-216-trace --lp-io-use-suffix=1 -- ../src/network-workloads/conf/modelnet-mpi-test-dfly-amg-216.conf

  The simulation runs in ROSS serial, conservative and optimistic modes.

   Note: Dragonfly and torus networks may have more number of nodes in the
   network than the number network traces (Some network nodes will only forward
   packets and they will not end up loading the traces). Thats why
   --num_net_traces argument is used to specify exact number of traces
   available in the DUMPI directory if there is a mis-match between number of
   network nodes and traces.

* Running the simulation in optimistic mode 
    

mpirun -np 4 ./install/bin/model-net-mpi-replay --sync=3
--num_net_traces=216 --workload_type="dumpi" --lp-io-dir=amg_216-trace
--lp-io-use-suffix=1
--workload_file=/home/df_traces/directory/dumpi-2014.03.03.15.09.03- 
-- src/network-workloads//conf/modelnet-mpi-test-dfly-amg-216.conf 

## Replaying Multiple Application Traces on Network Models
* Generate job allocation file (random or contiguous) using python scripts.

Allocation options 
1. Random allocation assigns a set of randomly selected network nodes to each job. 
1. Contiguous allocation assigns a set of contiguous network nodes to the jobs. 

See [generation CODES allocations](https://xgitlab.cels.anl.gov/codes/codes/wikis/codes-allocations) 
for instructions on how to generate job
allocation file using python. Example allocation files are in
src/network-workloads/conf/allocation-rand.conf, allocation-cont.conf.

* Run the simulation with multiple job allocations

./install/bin/model-net-mpi-replay --sync=1 --workload_conf_file=../src/network-workloads/workloads.conf --alloc_file=../src/network-workloads/conf/allocation-rand.conf --workload_type="dumpi" -- ../src/network-workloads/conf/modelnet-mpi-test-dfly-amg-216.conf

* To run in optimistic mode:

mpirun -np 4 ./install/bin/model-net-mpi-replay --sync=3 --workload_conf_file=../src/network-workloads/workloads.conf --alloc_file=allocation.conf --workload_type="dumpi" -- ../src/network-workloads/conf/modelnet-mpi-test-dfly-amg-216.conf

##  Running multiple MPI ranks (or cores) mapped to a compute node 
13- Update the config file to have multiple core/proc LPs mapped on to a
model-net LP. See example config file in:
`src/network-workloads/conf/modelnet-mpi-test-dfly-mul-cores.conf`

The overhead to send a message to the same node can be configured through the 
self_msg_overhead parameter in the config file (units are nanosecs). 
Its default value is 10 ns.

14- If running multiple MPI ranks per node with random allocations, the
allocation files must also be generated with multiple cores per node. See
[generating job allocations](https://xgitlab.cels.anl.gov/codes/codes/wikis/codes-allocations) for how to generate allocation files that use
multiple cores per node.

## Sampling and debugging options  

Runtime options can be used to enable time-stepped series data of simulation
with multiple workloads:

--**enable_sampling = 1** [Enables sampling of network & workload statistics after a specific simulated interval.
Default sampling interval is 5 millisec and default sampling end time is 3
secs. These values can be adjusted at runtime using --sampling_interval and
--sampling_end_time options.]

--**lp-io-dir-dir-name** [Turns on end of simulation statistics for dragonfly network model]

**NOTE: Each network model has information on how to run it with DUMPI Traces. For details, see [CODES network models](https://xgitlab.cels.anl.gov/codes/codes/wikis/codes-networks)**