hpc.Rmd

---
title: "High-Performance Computing (Argon Supercomputer)"
output: 
  html_document:
    code_folding: show
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(
  echo = TRUE,
  error = TRUE,
  comment = "",
  class.source = "fold-show")
```

# Overview

Analysis scripts for High Performance Computing (HPC) at the University of Iowa.
Currently uses Argon.

## Links to Documentation

Workflow: https://workflow.uiowa.edu/entry/new/3282/11927336

Cluster Systems Documentation:
https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76513411/Cluster+Systems+Documentation (archived at https://perma.cc/EKB8-ZKR7)

Argon Cluster Documentation:
https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76513466/Argon+Cluster (archived at https://perma.cc/6HST-VV6Y)

## Data Storage

Significant amounts of data storage are provided on Argon, but data are not backed up in any way unless special arrangements are made.
It is the responsibility of the user to back up important information.

## Sensitive Data

User agrees to refrain from storing Restricted data on HPC resources.
Data is classified as Restricted when the unauthorized disclosure, alteration or destruction of that data could cause a significant level of risk to the University or its affiliates.
Examples of Restricted data include data protected by state or federal privacy regulations and data pertaining to identified human subjects that has not been deidentified.

## Fees

At present there are no fees for the use of the Argon cluster for low-priority usage.
For large users, or those who want access to dedicated resources, the option of purchasing or renting supplemental system hardware may be available.
If you are interested in dedicated hardware, contact research-computing@uiowa.edu.

# Accessing Lab Drive

`/Shared/lss_itpetersen`

# Map Argon Drive to Local Computer

## Windows

1. Right click on `This PC` and select `Map Network Drive`
2. Select `Y` and type `\\data.hpc.uiowa.edu\argon_home\Documents`

# Accessing Argon

## SSH

`argon.hpc.uiowa.edu`

## Windows

Windows Explorer:

`\\data.hpc.uiowa.edu\argon_home` (username: itpetersen@uiowa.edu or hawkid@uiowa.edu)

Using SecureCRT for an SSH connection:
1. Download SecureCRT from UIowa Informational Technology Services: https://its.uiowa.edu/securecrt
2. In Hostname type `argon.hpc.uiowa.edu` and in username type HawkID
3. Click connect

## Mac

Mac OS Terminal:

`ssh itpetersen@argon.hpc.uiowa.edu` (on campus)

`ssh -p 40 itpetersen@argon.hpc.uiowa.edu` (off campus without VPN)

# General Commands
```
-  `pwd` print current working directory
-  `cd path/to/directory` sets working directory
-  `module load moduleName` loads modules required for analyses; necessary to complete prior to submitting jobs involving R
-  `module list` lists downloaded modules
-  `qsub` submit a job script to Argon processing
-  `R CMD BATCH` submit a single R script to Argon processing
Argon requires Linux-compatible file endings, which is problematic for files created outside Argon (using DOS or CRLF file endings). Use the following commands to check if your files are Argon compatible (i.e., Unix LF) and resolve incompatible file endings:
-   `file myfile.job | grep CRLF` checks if a file ending uses the incompatible CRLF format. Remember to check the ending of your file; this script assumes it is .job
-   `dos2unix myfile.job` converts CRLF file endings to Argon-compatible LF endings
-   `find ~/DirectoryName -name '*job' -exec dos2unix "{}" \;` converts all files in a directory that end with .job to LF format; change '*job' to change other file types
```

# Access Project Directories

```
cd /Users/itpetersen/Documents/Projects/Bayesian_IRT/
cd /Users/itpetersen/Documents/Projects/EXT_pilot/
cd /Users/itpetersen/Documents/Projects/Multiple_Imputation/
cd /Users/itpetersen/Documents/Projects/SelfRegulation_IRT/
cd /Users/itpetersen/Documents/Projects/Test/
```

# Linux

## Initial run

### Installing Linux Packages to Install `R` Packages

```bash
module load stack/2022.2
module load r/4.2.2_gcc-9.5.0
module load geos/3.9.1_gcc-9.5.0
module load gdal/2.4.4_gcc-9.5.0
module load proj/5.2.0_gcc-9.5.0
module load gsl/2.7.1_gcc-9.5.0
module load nlopt/2.7.0_gcc-9.5.0
module load jags/4.3.0_gcc-9.5.0-dev
module load zlib/1.2.13_gcc-9.5.0-dev
module load cmake/3.25.0_gcc-9.5.0
module load glpk/4.65_gcc-9.5.0-dev
module load libxml2/2.10.3_gcc-9.5.0
module load r-nloptr
module load r-zlibbioc/1.44.0_gcc-9.5.0
module load r-data-table/1.14.4_gcc-9.5.0
module load r-stringi/1.7.8_gcc-9.5.0
module load r-selectr/0.4-2_gcc-9.5.0
module load r-generics/0.1.3_gcc-9.5.0
module load r-fansi/1.0.3_gcc-9.5.0
module load r-utf8/1.2.2_gcc-9.5.0
module load r-pkgconfig/2.0.3_gcc-9.5.0
module load r-gtable/0.3.1_gcc-9.5.0
module load r-scales/1.2.1_gcc-9.5.0
module load r-tzdb/0.3.0_gcc-9.5.0
module load r-timechange/0.1.1_gcc-9.5.0
module load r-dbi/1.1.3_gcc-9.5.0

```

### Load the Latest Stack

`module load stack`

### Stack 2022.2

```bash
module load stack/2022.2
module load r/4.2.2_gcc-9.5.0
cd path
```

## Load Software Stacks

https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76513440/Argon+Software+List (archived at https://perma.cc/WJ4Q-GDUS)

`module load stack/2022.2`

## Job Script

```bash
#!/bin/sh

# Set working directory
cd /Users/itpetersen/Documents/Projects/Bayesian_IRT/

# Specify qsub options
#$ -pe smp 4
#$ -M isaac-t-petersen@uiowa.edu
#$ -m eas
#$ -l mf=512G
#$ -l h_vmem=512G
#$ -cwd
#$ -q UI-HM
#$ -e /Users/itpetersen/Documents/Projects/Bayesian_IRT/Output/
#$ -o /Users/itpetersen/Documents/Projects/Bayesian_IRT/Output/

# Load the environment modules
module load stack/2022.2
module load r/4.2.2_gcc-9.5.0

# Run the R script
Rscript ./Analyses/factorScores.R
```

### qsub options

`-pe smp 4`: specify a parellel environment and number of cores to be used (`smp` = shared memory parallel environment)
-  The `OMP_NUM_THREADS` variable is set to '1' by default.
If your code can take advantage of the threading then specify `OMP_NUM_THREADS` to be equal to the number of job cores per node requested.

`-M isaac-t-petersen@uiowa.edu`: Set the email address to receive email about jobs. This must be your University of Iowa email address.

`-m eas`: Specify when to send an email message (; ; ; ; )

-  `b` = beginning of job
-  `e` = end of job
-  `a` = when job is aborted
-  `s` = when job is suspended
-  `n` = no mail is sent

`-l mf=512G`: request a particular quantity of memory you expect to use (to be available for your computation to start; the request is only applicable at scheduling time. It is not a limit.)

`-l h_vmem=512G`: request a particular quantity of virtual memory you expect to use (to be available for your computation to start; the request is only applicable at scheduling time.
It is not a limit.)

`-cwd`: Determines whether the job will be executed from the current working directory.
If not specified, the job will be run from your home directory.

`-q UI-HM`: specify queue

`-e /Users/itpetersen/Documents/Projects/Bayesian_IRT/Output/`: Name of a file or directory for standard error.

`-o /Users/itpetersen/Documents/Projects/Bayesian_IRT/Output/`: Name of a file or directory for standard output.

## Submit Job

https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76513450/Basic+Job+Submission (archived at https://perma.cc/2SS2-LEJR)

https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76513452/Advanced+Job+Submission (archived at https://perma.cc/8H6G-2M2F)

`cd path/to/dataSet123`

[cd /Users/itpetersen/Documents/Projects/Test/Jobs]

`qsub myscript.job`

Job dependency (run `Job B` when `Job A` is finished):

`qsub -hold_jid JOB_ID test_B.job`

### Bayesian IRT

`cd /Users/itpetersen/Documents/Projects/Bayesian_IRT/Jobs`

`qsub bayesianIRT.job`

`qsub -hold_jid JOB_ID factorScores.job`

### Multiple Imputation

`cd /Users/itpetersen/Documents/Projects/Multiple_Imputation/Jobs`

`qsub srs_selfRegulation.R`

## Queues

https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76513468/Queues+and+Policies (archived at https://perma.cc/UUR7-XLBZ)

`UI`

`UI-HM`

`UI-GPU-HM`

`UI-DEVELOP`

`all.q`

## Queue Status

`qstat -g c -q QUEUE_NAME`

`qstat -g c -q UI`

## Job Status

`qstat -u itpetersen`

`qstat -j JOB_ID`

## Job Time and Memory

-  Terminated jobs: `qacct -j JOB_ID`
-  Jobs in progress: `qstat -j JOB_ID | grep usage`

## Cancel Job

`qdel JOB_ID`

## Run `R` Script

See [here](#runRscript)

# R

https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76514707/R+Programs+in+Batch+mode+for+HPC (archived at https://perma.cc/99JQ-43ZG)

## Find which versions of `R` are installed

`module spider R`

But, there may be more recent version of `R` installed in the "Additional Software Stacks" (https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76513440/Argon+Software+List; archived at https://perma.cc/WJ4Q-GDUS)

## Compile `R`

If you want to compile a more recent version of `R` than is available in the software stacks, see [here](https://brandonlebeau.org/2022/06/02/compile-hpc-iowa/) (archived at https://perma.cc/C6EX-EZL4).

## Install `R` packages

https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76514803/Adding+R+programs+to+a+personal+library (archived at https://perma.cc/3SRR-2JE7)

1. first load the `R` environment module
    - `module load r/4.2.2_gcc-9.5.0`
1. launch R
    - `R`
1. then install the package, (the code below uses a generic `package_name`):
    - `install.packages("package_name", repos = "http://cran.r-project.org", dependencies = TRUE, type = "source", Ncpus = 40)`
    - If using `packrat`, it is preferable to install packages by source, if possible, but you can remove `type = source` if you want to install binaries
    - Assuming that you do not have a personal library directory you will see

    ```
    Warning in install.packages("package_name", repos = "http://cran.r-project.org") :
    lib = "/opt/R/3.0.2/lib64/R/library"' is not writable

    Would you like to use a personal library instead?  (y/n)
    ```

1. Select `y`
1. Select `y` again when prompted to create the directory; your package should download and install into your personal library directory

Example:

```{r, eval = FALSE}
install.packages(c(
    "renv","psych","tidyverse","data.table","nlme","lme4","mirt","TeachingDemos","Amelia","mice","miceadds","abind","future","lavaan","blavaan","Rcpp","igraph","shinystan","StanHeaders","brms","rstan","rjags"),
    repos = "http://cran.r-project.org",
    dependencies = TRUE,
    Ncpus = 40)
```

### Install `R` package locally from source

`install.packages(path_to_file, repos = NULL, type = "source", Ncpus = 40)`

Example:

```{r, eval = FALSE}
install.packages(c(
    "renv","psych","tidyverse","data.table","nlme","lme4","mirt","TeachingDemos","Amelia","mice","miceadds","abind","future","lavaan","blavaan","Rcpp","igraph","shinystan","StanHeaders","brms","rstan","rjags"),
    repos = "http://cran.r-project.org",
    type = "source",
    dependencies = TRUE,
    Ncpus = 40)
```

### Managing packages for a project using `renv` {#renv}

1. Create a directory where you want you want to store the project.
1. `cd` to the above directory
1. Start R
1. Load the `renv` package: `library("renv")`
1. At the `R` prompt, initialize the `renv` project on the local repository of `R` packages with:
    - `renv::init(project = "/Users/itpetersen/Documents/Projects/SelfRegulation_IRT/")`
    - or `renv::init()` if you are in the intended working directory
1. You must restart your `R` session in the given project directory after running init in order for the changes to take effect!
1. From this point on you are working in a `renv` project.
Installed packages will go into a library within this project.
After initializing the `renv` project on the local repository of `R` packages, packages from the local repository can be installed with `renv::install()`:
    - `renv::install("package_name")`
1. To restart a `renv` project, simply start `R` from the directory created in step (1).
The project will initialize automatically.

To update version of `renv`: `renv::upgrade()`

To install packages: `renv::install("package_name")`

To update packages: `renv::update()`

To save the current state of your library: `renv::snapshot()`

To restore the state of your library from the lock file: `renv::restore()`

To disable `renv` on a project: `renv::deactivate()`

#### To control which packages to install with `renv` using a `DESCRIPTION` file

If you want to control which packages are installed in a `renv` project, you can use a `DESCRIPTION` file to specify the packages that should be installed:

1. Create a `DESCRIPTION` file in the project directory with the following format:

    ```
    Type: project
    Description: My project.
    Depends:
        packageName1,
        packageName2,
        packageName3
    ```

1. Run `renv::settings$snapshot.type("explicit")` to suppress dependency discover and to enable "explicit" mode: https://rstudio.github.io/renv/reference/dependencies.html#explicit-dependencies
1. Run `renv::init(bare=TRUE)` to initialize the project without attempting to discover and install `R` package dependencies

### To install a new package for a project after `renv` has been initialized

1. First, load the `R` environment module (see `Install R packages` section above)
1. cd to the directory of the `R` project
1. launch R
    - `R`
1. then install the package, (the code below uses a generic `package_name`):
    - `install.packages("package_name", repos = "http://cran.r-project.org", type = "source", dependencies = TRUE, Ncpus = 40)`
    - `install.packages(c("renv","psych","tidyverse","data.table","nlme","lme4","mirt","TeachingDemos","Amelia","mice","miceadds","abind","future","lavaan","blavaan","Rcpp","igraph","shinystan","StanHeaders","brms","rstan","rjags","renv"), repos = "http://cran.r-project.org", type = "source", dependencies = TRUE, Ncpus = 40)`
1. `renv::snapshot()`

## Update Packages

```{r, eval = FALSE}
update.packages(ask = FALSE)
```

## Run `R` Script {#runRscript}

`module load r/4.0.5_gcc-9.3.0`

`cd path/to/dataSet123`

[cd /Users/itpetersen/Documents/Projects/Test/Analyses]

`Rscript path/to/program.R`

[Rscript test.R]

## Quit R

`q()`

# Stan

https://github.com/stan-dev/rstan/wiki/Installing-RStan-on-Linux (archived at https://perma.cc/89H9-L8S6)

Makevars:

`\\data.hpc.uiowa.edu\argon_home\.R\Makevars`

# Troubleshooting

## Having trouble installing packages?

Sometimes Argon will fail to install packages in an R workspace.
We advise the following steps:

1. Sometimes R provides an error stating something like "fatal error: modulename: No such file or directory".
In this instance, you may wish to exist R with q() and load all the modules that begin with the listed module name, as listed above in "Installing Linux Packages to Install R Packages".
Then reopen R and try to install the package again.
1. You can also try installing the package from binary using the RStudio Package Manager (RSPM):

    ```r
    remotes::install_github("cran4linux/rspm")
    rspm::enable()
    install.packages("PACKAGE_NAME")
    ```

1. If that does not work, you can try downloading the .tar file directly from the CRAN repository. Then copy the file into your Argon folder and type 'install.packages("/Users/path/to/directory/package_name", repos = NULL, type = "source")'.
Note that you may need to download an older version of the package (e.g., from the [CRAN Archive](https://cran.r-project.org/src/contrib/Archive/)), such that it is compatible with the version of R you are running on Argon (which it typically not the most recent R version):

    ```r
    install.packages("http://cran.r-project.org/src/contrib/Archive/MASS/MASS_7.3-60.0.1.tar.gz", repos = NULL, type = "source")
    ```

1. If steps 1 and 2 fail, email research-computing@uiowa.edu with the error, asking for help to figure out how to install the packages.

## Error When Installing Packages from Source

You may have to set environment variables in each module file to help the compiler find headers and libraries.
Note that if you run into any C++ code, you will need to set the `CPLUS_INCLUDE_PATH` variable.

### `nloptr`

1. Run the following commands

```bash
module load stack/2022.1
module load r/4.1.3_gcc-9.4.0
module load nlopt
```

2. Set `LIBRARY_PATH` so linker can find library while launching `R` session (single line below):

```
LIBRARY_PATH=$ROOT_NLOPT/lib64:$LIBRARY_PATH R
```

3. In `R` console, install `nloptr` (two lines below)

```{r, eval = FALSE}
install.packages(verbose = 1, "nloptr")
```

### `tkrplot`

1. Run the following commands

```bash
module load stack/2022.1
module load r/4.1.3_gcc-9.4.0
```

2. Set LIBRARY_PATH so linker can find library while launching `R` session (single line below):

```
C_INCLUDE_PATH=$ROOT_XPROTO/include LIBRARY_PATH=$ROOT_LIBXEXT/lib:$ROOT_LIBXSCRNSAVER/lib R
```

3. In `R` console, install `tkrplot` (two lines below)

```{r, eval = FALSE}
install.packages(verbose = 1, "tkrplot")
```

## Error Reading from Connection

```
Error in unserialize(node$con) : error reading from connection
Calls: parlmice ... FUN -> recvData -> recvData.SOCK0node -> unserialize
```

There likely wasn't sufficient memory for a given core.
Try increasing the max memory available and decreasing the number of cores and/or slots, so there is more memory available per core:

https://stackoverflow.com/questions/46186375/r-parallel-error-in-unserializenodecon-in-hpc (archived at https://perma.cc/MF6V-NAVS)

https://stackoverflow.com/questions/17015598/error-calling-serialize-r-function (archived at https://perma.cc/3Q75-DA2D)

https://gforge.se/2015/02/how-to-go-parallel-in-r-basics-tips/#Memory_load (archived at https://perma.cc/2JRF-8Y5F)

# Appendix {#appendix}

## Old Stacks

### Stack 2022.1

```bash
module load stack/2022.1
module load r/4.1.3_gcc-9.4.0
cd path
```

### Stack 2021.1

```bash
module load stack/2021.1
module load r/4.0.5_gcc-9.3.0
cd path
```

### Stack 2020.2

```bash
module load stack/2020.2
module load r/4.0.2_gcc-8.4.0
cd path
```

### Stack 2020.1

```bash
module load stack/2020.1
module load r/3.6.2_gcc-9.2.0
cd path
```

## Managing packages for a project using `packrat`

Please note: we now use [`renv`](#renv) rather than `packrat` for package management

https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76514803/Adding+R+programs+to+a+personal+library (archived at https://perma.cc/3SRR-2JE7)

1. Create a directory where you want you want to store the project.
1. `cd` to the above directory
1. Start R
1. Load the `packrat` package: `library("packrat")`
1. At the `R` prompt, initialize the `packrat` project on the local repository of `R` packages with: `packrat::init(project = "/Users/itpetersen/Documents/Projects/INSERT_PROJECT_NAME/", options = list(local.repos = "/Users/itpetersen/R/x86_64-pc-linux-gnu-library/4.0"))`
1. You must restart your `R` session in the given project directory after running init in order for the changes to take effect!
1. From this point on you are working in a `packrat` project.
Installed packages will go into a library within this project.
After initializing the `packrat` project on the local repository of `R` packages, packages from the local repository can be installed with `packrat::install_local()`:
    - `packrat::install_local("package_name")`
1. To restart a packrat project simply start `R` from the directory created in step (1). The project will initialize automatically.

To save the current state of your library: `packrat::snapshot()`; if that command fails due to an error when fetching sources, try `packrat::snapshot(snapshot.sources = FALSE)` 

To disable `packrat` on a project: `disable(restart = FALSE)`

### To install a new package for a project after `packrat` has been initialized

1. First, load the `R` environment module (see `Install R packages` section above)
1. cd to the directory of the `R` project
1. launch R
    - `R`
1. then install the package, (the code below uses a generic `package_name`):
    - `install.packages("package_name", repos = "http://cran.r-project.org", type = "source", dependencies = TRUE, Ncpus = 40)`
    - `install.packages(c("packrat","psych","tidyverse","data.table","nlme","lme4","mirt","TeachingDemos","Amelia","mice","miceadds","abind","future","lavaan","blavaan","Rcpp","igraph","shinystan","StanHeaders","brms","rstan","rjags","renv"), repos = "http://cran.r-project.org", type = "source", dependencies = TRUE, Ncpus = 40)`
1. `packrat::snapshot()`; if that command fails due to an error when fetching sources, try `packrat::snapshot(snapshot.sources = FALSE)`

### Build `packrat` environment on a compute node (only if necessary if package load fails due to issues building packages)

1. `qlogin` -- for more info, see here: https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76513454/Qlogin+for+Interactive+Sessions (archived at https://perma.cc/Y3A9-WQ3W)
1. `module load stack/2020.2-base_arch` -- this will ensure that the modules point to the lowest common multiarchitecture on Argon and will run on all nodes
1. `module load r`
1. cd to the directory of the `R` project
1. `R`
1. Create `packrat` project (see above)
1. Install packages
1. Take a package snapshot using `packrat`
1. `exit` -- to exit the qlogin session

Note: Your `packrat` environment will then be linked to the proper `glpk` library and will run on any Argon node.
You do not need to use the `2020.2-base_arch` module at run time, only build time.