Skip to content

CosmoLike/cocoa

Repository files navigation

Screenshot 2024-09-16 at 10 55 14 PM

Table of contents

  1. Overview of the Cobaya-CosmoLike Joint Architecture (Cocoa)
  2. Installation of core packages via Conda
  3. Installation and Compilation of external modules
  4. Running Examples
  5. Creating Cosmolike projects (external readme)
  6. Appendix
    1. Credits
    2. Additional Installation Notes
    3. FAQ: What if installation or compilation goes wrong?
    4. FAQ: How do we compile the Boltzmann, CosmoLike, and Likelihood codes separately
    5. FAQ: How do we run cocoa on a laptop? The docker image named whovian-cocoa
    6. FAQ: How do we use an available Anaconda module on HPC?
    7. FAQ: What if there is no Conda? Miniconda installation
    8. FAQ: How do we make the Slow/Fast decomposition on MCMC chains with Cosmolike? Manual Blocking
    9. FAQ: How do we switch Cocoa's adopted CAMB/CLASS/Polychord? (external readme)
    10. FAQ: How do we download modern CMB data? (external readme)
    11. FAQ: How do we set the environment for Machine Learning projects?
    12. FAQ: How can users improve their Bash/C/C++ knowledge to develop Cocoa/Cosmolike?
    13. Warning about Weak Lensing YAML files in Cobaya
    14. FAQ: How do we install Cocoa without conda?
    15. FAQ: How do we push changes to the Cocoa main branch? A few git hacks
    16. FAQ: How do we develop from a git tag? A few more git hacks
    17. FAQ: How do we download and run Cosmolike projects?

Overview of the Cobaya-CosmoLike Joint Architecture (Cocoa)

Cocoa allows users to run CosmoLike routines inside the Cobaya framework. CosmoLike can analyze data primarily from the Dark Energy Survey and simulate future multi-probe analyses for LSST and Roman Space Telescope.

Besides integrating Cobaya and CosmoLike, Cocoa introduces shell scripts that allow users to containerize Cobaya, the Boltzmann codes and multiple likelihoods. The container structure ensures that users can adopt consistent versions for the Fortran/C/C++ compilers and libraries across multiple machines; this greatly simplifies debugging.

Our scripts never install packages and Python modules on $HOME/.local, where $HOME is a shell environment variable that points to the user's home folder, as that would make them global to the user. Such behavior would interfere with all previously installed packages, potentially creating dependency incompatibilities between different projects the user works on. This behavior enables users to work on multiple instances of Cocoa simultaneously, similar to what was possible with CosmoMC.

This readme file presents basic and advanced instructions for installing all Cobaya and CosmoLike components.

Installation of core packages via Conda

Step 1️⃣: Download the file cocoapy39.yml yml file

wget https://raw.githubusercontent.com/CosmoLike/cocoa/dev/cocoapy39.yml

create the cocoa environment,

conda env create --name cocoa --file=cocoapy39.yml

activate it

conda activate cocoa

and create symbolic links that will give better names for the GNU compilers

ln -s "${CONDA_PREFIX}"/bin/x86_64-conda_cos6-linux-gnu-gcc "${CONDA_PREFIX}"/bin/gcc
ln -s "${CONDA_PREFIX}"/bin/x86_64-conda_cos6-linux-gnu-g++ "${CONDA_PREFIX}"/bin/g++
ln -s "${CONDA_PREFIX}"/bin/x86_64-conda_cos6-linux-gnu-gfortran "${CONDA_PREFIX}"/bin/gfortran
ln -s "${CONDA_PREFIX}"/bin/x86_64-conda-linux-gnu-gcc-ar "${CONDA_PREFIX}"/bin/gcc-ar
ln -s "${CONDA_PREFIX}"/bin/x86_64-conda-linux-gnu-gcc-ranlib "${CONDA_PREFIX}"/bin/gcc-ranlib
ln -s "${CONDA_PREFIX}"/bin/x86_64-conda-linux-gnu-ld "${CONDA_PREFIX}"/bin/ld

Users can now proceed to step 2️⃣.

Tip

To install the Cocoa environment on a supercomputer, users may take advantage of the fact that many HPC environments provide the Anaconda installer as an external module. If this is the case, check the Appendix FAQ: How do we use an available Anaconda module on HPC?.

Tip

If the user is not working on an HPC environment that offers Anaconda or Miniconda, check the Appendix FAQ: What if there is no Conda? Miniconda installation.

Tip

We provide the YML file cocoapy310.yml so users can work on Python 3.10. Users must also modify the following flag on set_installation_options.sh before proceeding further with the Cocoa installation.

[Adapted from Cocoa/set_installation_options.sh shell script] 
# ------------------------------------------------------------------------------
# Adopted Python version -------------------------------------------------------
# ------------------------------------------------------------------------------
export PYTHON_VERSION=3.9

Note

Core packages include compilers and numerical libraries (e.g., GSL and FFTW) that we need but will certainly never develop/modify.

Step 2️⃣: Install git-lfs when loading the Conda cocoa environment for the first time.

git-lfs install

Warning

In the next section, we assume the user loaded the Cocoa conda environment via the conda activate cocoa command.

Installation and Compilation of external modules

Step 1️⃣: Download Cocoa's latest release and go to the cocoa main folder,

"${CONDA_PREFIX}"/bin/git clone --depth 1 https://github.com/CosmoLike/cocoa.git --branch v4.0-beta14 cocoa

and

cd ./cocoa/Cocoa

Users can now proceed to step 2️⃣.

Tip

If the user wants to clone the repository in development mode, type the following command to clone the repository from the latest main commit.

"${CONDA_PREFIX}"/bin/git clone [email protected]:CosmoLike/cocoa.git cocoa

Tip

If the developer wants to start coding a new feature or fix a bug from a git tag, check the appendix FAQ: How do we push changes to the Cocoa main branch? A few git hacks subsection How to develop starting from a git tag?

Step 2️⃣: Run the script setup_cocoa.sh via

source setup_cocoa.sh

Note

This script downloads and decompresses external modules set on the set_installation_options.sh script (e.g., CAMB and Class).

Step 3️⃣: Run the script compile_cocoa.sh by typing

source compile_cocoa.sh

This script compiles external modules set on the set_installation_options.sh script (e.g., CAMB and Class).

Tip

Cocoa does not install many external modules by default, but users may find them helpful in a particular project. Check the many available options on the set_installation_options.sh shell script in this case. Then, rerun steps 2️⃣ and 3️⃣.

Running Examples

We assume that you are still in the Conda cocoa environment from the previous conda activate cocoa command and that you are in the cocoa main folder cocoa/Cocoa,

Step 1️⃣: Activate the private Python environment by sourcing the script start_cocoa.sh

source start_cocoa.sh

Note

Users will see a terminal like this: $(cocoa)(.local). This is a feature, not a bug!

Step 2️⃣: Select the number of OpenMP cores (below, we set it to 8).

export OMP_PROC_BIND=close; export OMP_NUM_THREADS=8

Note

The environmental variable OMP_PROC_BIND, when set to close, places the OpenMP cores as closely as possible (in the same chiplet). This setting is important as current architectures limit communications bandwidth between different chiplets (e.g., the cores inside an AMD 96 cores processor are scattered on a dozen chiplets).

Examples not involving Cosmolike

Step 3️⃣: The folder projects/example contains a dozen examples involving different likelihoods. So, run the cobaya-run on the first example following the commands below.

One model evaluation:

mpirun -n 1 --oversubscribe --mca btl vader,tcp,self --bind-to core:overload-allowed --rank-by core --map-by numa:pe=${OMP_NUM_THREADS} cobaya-run  ./projects/example/EXAMPLE_EVALUATE1.yaml -f

MCMC:

mpirun -n 4 --oversubscribe --mca btl vader,tcp,self --bind-to core:overload-allowed --rank-by core --map-by numa:pe=${OMP_NUM_THREADS} cobaya-run ./projects/example/EXAMPLE_MCMC1.yaml -f

Examples involving Cosmolike

Step 3️⃣: The folder projects/lsst_y1 contains a dozen examples involving different combinations of two-point correlation functions. So, run the cobaya-run on the first example following the commands below.

One model evaluation:

mpirun -n 1 --oversubscribe --mca btl vader,tcp,self --bind-to core:overload-allowed --rank-by core --map-by numa:pe=${OMP_NUM_THREADS} cobaya-run ./projects/lsst_y1/EXAMPLE_EVALUATE1.yaml -f

MCMC:

mpirun -n 4 --oversubscribe --mca btl vader,tcp,self --bind-to core:overload-allowed --rank-by core --map-by numa:pe=${OMP_NUM_THREADS} cobaya-run ./projects/lsst_y1/EXAMPLE_MCMC1.yaml -f

Tip

To run Jupyter Notebook, assuming cocoa is installed on a local machine, type, after step 2️⃣, the command

jupyter notebook --no-browser --port=8888

The terminal will show a message similar to the following template:

(...)
[... NotebookApp] Jupyter Notebook 6.1.1 is running at:
[... NotebookApp] http://f0a13949f6b5:8888/?token=XXX
[... NotebookApp] or http://127.0.0.1:8888/?token=XXX
[... NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

Now go to the local internet browser and type http://127.0.0.1:8888/?token=XXX, where XXX is the previously saved token displayed on the line

[... NotebookApp] or http://127.0.0.1:8888/?token=XXX

The project lsst-y1 contains jupyter notebook examples located at projects/lsst_y1.

Tip

To run Jupyter Notebook, assuming cocoa is installed on a remote machine with open port 8888, type, after step 2️⃣, the command

jupyter notebook --no-browser --port=8888 --ip=XX.YY.ZZ.123

where XX.YY.ZZ.123 should be replaced by the server IP. The terminal will show a message similar to the following template:

(...)
[... NotebookApp] Jupyter Notebook 6.1.1 is running at:
[... NotebookApp] http://XX.YY.ZZ.123:8888/?token=XXX
[... NotebookApp] or http://127.0.0.1:8888/?token=XXX
[... NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

Now go to the local internet browser and type http://XX.YY.ZZ.123:8888/?token=XXX.

Tip

If the user wants to download a project not provided by default or intends to clone existing projects in development mode, check the Appendix FAQ: How do we download and run Cosmolike projects?.

Appendix

Credits

The following is not an exhaustive list of the codes we use/download/adopt

  • Cobaya is a framework developed by Dr. Jesus Torrado and Prof. Anthony Lewis

  • Cosmolike is a framework developed by Prof. Elisabeth Krause and Prof. Tim Eifler

  • CAMB is a Boltzmann code developed by Prof. Anthony Lewis

  • CLASS is a Boltzmann code developed by Prof. Julien Lesgourgues and Dr. Thomas Tram

  • Polychord is a sampler code developed by Dr. Will Handley, Prof. Lasenby, and Prof. M. Hobson

  • CLIK is the likelihood code used to analyze Planck and SPT data, maintained by Prof. Karim Benabed

  • SPT is the official likelihood of the South Pole Telescope 3G Year 1

  • MFLike is the official likelihood of the Simons Observatory

  • ACTLensing is the official lensing likelihood of the ACT collaboration developed by Prof. Mathew Madhavacheril

  • HiLLiPoP CMB likelihood is a multifrequency CMB likelihood for Planck data.

  • Lollipop CMB likelihood is a Planck low-l polarization likelihood.

Following best practices, Cocoa scripts download most external modules from their original repositories, including Cobaya, CAMB, Class, Polychord, ACT-DR6, HiLLiPoP, and Lollipop. We do not want to discourage people from cloning code from their original repositories. Our repository has included a few likelihoods as compressed xz file format. The work of those authors is extraordinary, and users must cite them appropriately.

Additional Installation Notes

  • Installation of core packages via Conda

Tip

For those working on projects that utilize machine-learning-based emulators, the Appendix Setting-up conda environment for Machine Learning emulators provides additional commands for installing the necessary packages.

Tip

We provide a docker image named whovian-cocoa that facilitates cocoa installation on Windows and MacOS. For further instructions, refer to the Appendix FAQ: How do you run cocoa on your laptop? The docker container is named whovian-cocoa.

Note

The conda installation method should be chosen in most cases. In the rare instances in which the user cannot work with Conda, refer to the Appendix Installation of Cocoa's core packages without Conda, as it contains instructions for a much slower (and prone to errors) but conda-independent installation method.

  • Installation and Compilation of external modules

Tip

If the user wants to compile only a subset of these packages, refer to the appendix Compiling Boltzmann, CosmoLike, and Likelihood codes separately.

Note

Our scripts never install packages on $HOME/.local as that would make them global to the user. All requirements for Cocoa are installed at

 Cocoa/.local/bin
 Cocoa/.local/include
 Cocoa/.local/lib
 Cocoa/.local/share

This behavior enables users to work on multiple instances of Cocoa simultaneously, similar to what was possible with CosmoMC.

  • Running Examples

Tip

We offer the flag COCOA_RUN_EVALUATE as an alias (syntax-sugar) for mpirun -n 1 --oversubscribe --mca btl vader,tcp,self --bind-to core:overload-allowed --rank-by core --map-by numa:pe=4 cobaya-run.

Tip

We offer the flag COCOA_RUN_MCMC as an alias (syntax-sugar) for mpirun -n 4 --oversubscribe --mca btl vader,tcp,self --bind-to core:overload-allowed --rank-by core --map-by numa:pe=4 cobaya-run.

Note

Additional explanations about our mpirun flags: Why the --mca btl vader,tcp,self flag? Conda-forge developers don't compile OpenMPI with Infiniband compatibility.

Note

Additional explanations about our mpirun flags: Why the --bind-to core:overload-allowed --map-by numa:pe=${OMP_NUM_THREADS} flag? This flag enables efficient hybrid MPI + OpenMP runs on NUMA architecture.

Note

Additional explanations about the functioning start_cocoa.sh/stop_cocoa.sh scripts: why two separate shell environments, (cocoa) and (.local)? Users should be able to manipulate multiple Cocoa instances seamlessly, which is particularly useful when running chains in one instance while experimenting with code development in another. Consistency of the environment across all Cocoa instances is crucial, and the start_cocoa.sh/stop_cocoa.sh scripts handle the loading and unloading of environmental path variables.

⁉️ FAQ: What if installation or compilation goes wrong?

  • The script set_installation_options script contains a few additional flags that may be useful. Some of these flags are shown below:

    [Adapted from Cocoa/set_installation_options.sh shell script] 
    
    # ------------------------------------------------------------------------------
    # VERBOSE AS DEBUG TOOL --------------------------------------------------------
    # ------------------------------------------------------------------------------
    #export COCOA_OUTPUT_VERBOSE=1
    
    # ------------------------------------------------------------------------------
    # If set, COSMOLIKE will compile with DEBUG flags ------------------------------
    # ------------------------------------------------------------------------------
    #export COSMOLIKE_DEBUG_MODE=1
    
    # ------------------------------------------------------------------------------
    # The flags below allow users to skip downloading specific datasets ------------
    # ------------------------------------------------------------------------------
    # export IGNORE_BICEP_CMB_DATA=1
    # export IGNORE_HOLICOW_STRONG_LENSING_DATA=1
    # export IGNORE_SN_DATA=1
    # export IGNORE_BAO_DATA=1
    # export IGNORE_SPT_CMB_DATA=1
    export IGNORE_SIMONS_OBSERVATORY_CMB_DATA=1
    # export IGNORE_PLANCK_CMB_DATA=1
    export IGNORE_CAMSPEC_CMB_DATA=1
    export IGNORE_LIPOP_CMB_DATA=1
    
    (...)
    
    # ------------------------------------------------------------------------------
    # The keys below control which packages will be installed and compiled 
    # ------------------------------------------------------------------------------
    #export IGNORE_CAMB_CODE=1
    #export IGNORE_CLASS_CODE=1
    #export IGNORE_COSMOLIKE_CODE=1
    #export IGNORE_POLYCHORD_SAMPLER_CODE=1
    #export IGNORE_PLANCK_LIKELIHOOD_CODE=1
    #export IGNORE_ACTDR4_CODE=1
    #export IGNORE_ACTDR6_CODE=1
    #export IGNORE_CPP_CUBA_INSTALLATION=1
    #export IGNORE_VELOCILEPTORS_CODE=1
    #export IGNORE_SIMONS_OBSERVATORY_LIKELIHOOD_CODE=1
    #export IGNORE_CAMSPEC_LIKELIHOOD_CODE=1
    #export IGNORE_LIPOP_LIKELIHOOD_CODE=1
    
    (...)
    

Steps to debug Cocoa

  • The first step is to define the COCOA_OUTPUT_VERBOSE and COSMOLIKE_DEBUG_MODE flags to obtain a more detailed output. To accomplish that, we advise users to uncomment the lines below that are part of the set_installation_options.sh script and then restart the cocoa private environment by running source stop_cocoa.sh; source start_cocoa.sh

    [Adapted from Cocoa/set_installation_options.sh shell script] 
    
    # ------------------------------------------------------------------------------
    # VERBOSE AS DEBUG TOOL --------------------------------------------------------
    # ------------------------------------------------------------------------------
    export COCOA_OUTPUT_VERBOSE=1
    
    # ------------------------------------------------------------------------------
    # If set, COSMOLIKE will compile with DEBUG flags ------------------------------
    # ------------------------------------------------------------------------------
    export COSMOLIKE_DEBUG_MODE=1
    
    (....)
    
  • The second step consists of rerunning the particular script that failed with the verbose output set. The scripts setup_cocoa.sh and compile_cocoa.sh run many shell scripts. Users may find it advantageous to run only the routine that failed. For further information on how to do that, see the appendix FAQ: How do we compile the Boltzmann, CosmoLike, and Likelihood codes separately.

After fixing a particular issue, users should rerun the shell scripts setup_cocoa.sh and compile_cocoa.sh to ensure all packages are installed and compiled correctly.

⁉️ FAQ: How do we compile the Boltzmann, CosmoLike, and Likelihood codes separately

To avoid excessive compilation or download times during development, users can use scripts located at Cocoa/installation_scripts/ that compile only a specific module or download only a particular dataset. A few examples of these scripts are:

 $(cocoa)(.local) cd "${ROOTDIR:?}"
 $(cocoa)(.local) source "${ROOTDIR:?}"/installation_scripts/compile_act_dr4.sh
 $(cocoa)(.local) source "${ROOTDIR:?}"/installation_scripts/compile_camb.sh
 $(cocoa)(.local) source "${ROOTDIR:?}"/installation_scripts/compile_class.sh
 $(cocoa)(.local) source "${ROOTDIR:?}"/installation_scripts/compile_planck.sh
 $(cocoa)(.local) source "${ROOTDIR:?}"/installation_scripts/compile_polychord.sh

Above and below, the $(cocoa)(.local) emphasizes they should run after activating the cocoa environments. The shell subroutines that download external modules from their original Git repositories are shown below.

 $(cocoa)(.local) cd "${ROOTDIR:?}"
 $(cocoa)(.local) source "${ROOTDIR:?}"/installation_scripts/setup_act_dr4.sh
 $(cocoa)(.local) source "${ROOTDIR:?}"/installation_scripts/setup_camb.sh
 $(cocoa)(.local) source "${ROOTDIR:?}"/installation_scripts/setup_class.sh
 $(cocoa)(.local) source "${ROOTDIR:?}"/installation_scripts/setup_polychord.sh
 $(cocoa)(.local) source "${ROOTDIR:?}"/installation_scripts/setup_cobaya.sh

To ensure these scripts can download and install these packages, users must be sure that the environment keys below are NOT set. These keys are shown on set_installation_options.sh. The command unset -v unset them.

 unset -v IGNORE_CAMB_CODE
 unset -v IGNORE_CLASS_CODE
 unset -v IGNORE_POLYCHORD_SAMPLER_CODE
 unset -v IGNORE_PLANCK_LIKELIHOOD_CODE
 unset -v IGNORE_ACTDR4_CODE
 unset -v IGNORE_COBAYA_CODE

Below, we show the shell subroutines that download and unpack data from multiple experiments.

 $(cocoa)(.local) cd "${ROOTDIR:?}"
 $(cocoa)(.local) source "${ROOTDIR:?}"/installation_scripts/unxv_act_dr6.sh
 $(cocoa)(.local) source "${ROOTDIR:?}"/installation_scripts/unxv_bao.sh
 $(cocoa)(.local) source "${ROOTDIR:?}"/installation_scripts/unxv_bicep.sh
 $(cocoa)(.local) source "${ROOTDIR:?}"/installation_scripts/unxv_camspec.sh
 $(cocoa)(.local) source "${ROOTDIR:?}"/installation_scripts/unxv_h0licow.sh
 $(cocoa)(.local) source "${ROOTDIR:?}"/installation_scripts/unxv_lipop.sh
 $(cocoa)(.local) source "${ROOTDIR:?}"/installation_scripts/unxv_planck2018_basic.sh
 $(cocoa)(.local) source "${ROOTDIR:?}"/installation_scripts/unxv_simons_observatory.sh
 $(cocoa)(.local) source "${ROOTDIR:?}"/installation_scripts/unxv_sn.sh
 $(cocoa)(.local) source "${ROOTDIR:?}"/installation_scripts/unxv_spt.sh

To ensure these scripts can download these datasets, users must be sure that the environment keys below are NOT set. These keys are shown on set_installation_options.sh. The command unset -v unset them.

 unset -v IGNORE_ACTDR6_DATA
 unset -v IGNORE_BAO_DATA
 unset -v IGNORE_BICEP_CMB_DATA
 unset -v IGNORE_CAMSPEC_CMB_DATA
 unset -v IGNORE_HOLICOW_STRONG_LENSING_DATA
 unset -v IGNORE_LIPOP_CMB_DATA
 unset -v IGNORE_PLANCK_CMB_DATA
 unset -v IGNORE_SIMONS_OBSERVATORY_CMB_DATA
 unset -v IGNORE_SN_DATA
 unset -v IGNORE_SPT_CMB_DATA

⁉️ FAQ: How do we run cocoa on a laptop? The docker image named whovian-cocoa

We provide the docker image whovian-cocoa to facilitate the installation of Cocoa on Windows and MacOS. This appendix assumes the users already have the docker engine installed on their local PC. For instructions on installing the docker engine in specific operating systems, please refer to Docker's official documentation.

Step 1️⃣: Create a folder and go to the location on the host computer that you want to provide access to the Docker container, as shown below.

 mkdir -p cocoa_docker
 cd ./cocoa_docker

Note

The flag -v $(pwd):/home/whovian/host/ in the docker run command ensures that files on the host computer have been mounted to the directory /home/whovian/host/. Files within the folder where the Docker container was initialized are accessible at /home/Whovian/host/. Users should work inside this directory to avoid losing work if the docker image needs to be deleted.

Warning

Do not run the Docker container on a general folder (like the host's home directory); this would provide too much access to the Docker container. Accidents happen, especially when dealing with dangerous bash commands such as rm (deletion).

Step 2️⃣: Download the docker image whovian-cocoa, name the associated container cocoa2023, and run the container for the first time, type:

docker run --platform linux/amd64 --hostname cocoa --name cocoa2023 -it -p 8888:8888 -v $(pwd):/home/whovian/host/ -v ~/.ssh:/home/whovian/.ssh:ro vivianmiranda/whovian-cocoa

Following the command above, users should see the following text on the screen terminal:

Screenshot 2023-12-19 at 1 26 50 PM

Step 3️⃣: Init Conda when running the container the first time, as shown below.

conda init bash
source ~/.bashrc
conda activate cocoa
git-lfs install

Step 4️⃣: Access the host computer on /home/whovian/host/

cd ~/host/

Now proceed with the standard cocoa installation in section Installation and Compilation of external modules

Once installation is complete, the user must learn how to start, use, and exit the container. Below, we answer a few common questions about using/managing Docker containers.

Tip

Assuming the user maintained the container name cocoa2023 via the flag --name cocoa2023 on the docker run command, type:

 docker start -ai cocoa2023

to restart the container after the first exit.

Tip

To run Jupyter Notebooks within the whovian-cocoa docker container installed on a local machine, type the following command:

 jupyter notebook --no-browser --port=8888

The terminal will show a message similar to the following template:

 [... NotebookApp] Writing notebook server cookie secret to /home/whovian/.local/share/jupyter/runtime/notebook_cookie_secret
 [... NotebookApp] WARNING: The notebook server is listening on all IP addresses and not using encryption. This is not recommended.
 [... NotebookApp] Serving notebooks from local directory: /home/whovian/host
 [... NotebookApp] Jupyter Notebook 6.1.1 is running at:
 [... NotebookApp] http://f0a13949f6b5:8888/?token=XXX
 [... NotebookApp] or http://127.0.0.1:8888/?token=XXX
 [... NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

Now go to the local internet browser and type http://127.0.0.1:8888/?token=XXX, where XXX is the previously saved token displayed on the line

[... NotebookApp] or http://127.0.0.1:8888/?token=XXX  

Tip

To run the Jupyter Notebook on the whovian-cocoa docker container installed in a remote server, adjust the command below

ssh your_username@your_sever.com -L 8888:localhost:8888

before typing http://127.0.0.1:8888/?token=XXX on the local desktop/laptop. This will bind the server port 8888 to the local port 8888.

[... NotebookApp] or http://127.0.0.1:8888/?token=XXX  

⁉️ FAQ: How do we use an available Anaconda module on HPC?

Below, we list users' most common issues when installing Cocoa conda environments in a supercomputer environment using a globally defined Anaconda module.

  • ⁉️ Conda command not found.

Anaconda is not usually set by default on HPC environments but may be available as a module. For example, on the Midway HPC cluster, it can be loaded using the command below.

module load Anaconda3/2022.10

If you are not sure about the name of the available Anaconda module, type the command

module avail An

to show all modules with names that start with An. The output should resemble the following.

Anaconda
  • ⁉️ Installation seems to take forever.

There are various reasons why installing the Cocoa conda environment may take a long time. Here is a checklist of good practices to overcome this problem.

1️⃣ Never install conda environments using the login node.

Instead, request an interactive job with a few cores. However, users must know that some supercomputers do not provide internet access on computing nodes. Ask the HPC staff for a queue dedicated to installing and compiling code; it should exist in a well-designed HPC environment.

  • ⁉️ Conda installation is interrupted due to quota limitations.

Supercomputers usually enforce strict quota limits on home folders. These limits apply to the total file size and the number of files. By default, Anaconda modules install new environments at $HOME/.conda/envs. Anaconda also stores Gigabytes of downloaded packages in the $HOME/.conda/pkgs folder; the pkgs folder is used by Anaconda as a package cache.

1️⃣ Create an Anaconda folder in a project folder outside $HOME with significantly more tolerant quota restrictions. For instance, we used the command below on the Midway supercomputer to create an Anaconda folder in the KICP projects partition.

mkdir /project2/kicp/XXX/anaconda/

2️⃣ Set the pkgs package cache folder to anaconda/pkgs.

conda config --add pkgs_dirs /project2/kicp/XXX/anaconda/pkgs

3️⃣: Set the env folder to anaconda/envs/

conda config --add envs_dirs /project2/kicp/XXX/anaconda/envs

4️⃣ Set the following flags for a proper conda environment installation

conda config --set auto_update_conda false
conda config --set show_channel_urls true
conda config --set auto_activate_base false
conda config --prepend channels conda-forge
conda config --set channel_priority strict
conda init bash

5️⃣ Source the .bashrc file so the new Anaconda settings are loaded.

source ~/.bashrc

6️⃣ After completing steps 1️⃣-5️⃣, check the $HOME/.condarc file with the command

more $HOME/.condarc

to make sure it resembles the one below.

auto_update_conda: false
show_channel_urls: true
auto_activate_base: false
channels:
  - conda-forge
  - defaults
channel_priority: strict
verbosity: 0
pkgs_dirs:
  - /project2/kicp/XXX/anaconda/pkgs
envs_dirs:
  - /project2/kicp/XXX/anaconda/envs/

⁉️ FAQ: What if there is no Conda? Miniconda installation

Step 1️⃣: Download and run the Miniconda installation script.

export CONDA_DIR="/gpfs/home/XXX/miniconda"

mkdir "${CONDA_DIR:?}"

wget https://repo.continuum.io/miniconda/Miniconda3-py38_23.9.0-0-Linux-x86_64.sh

/bin/bash Miniconda3-py38_23.9.0-0-Linux-x86_64.sh -f -b -p "${CONDA_DIR:?}"

Please don't forget to adapt the path assigned to CONDA_DIR in the command above:

Step 2️⃣: After installation, users must source the conda configuration file, as shown below:

source $CONDA_DIR/etc/profile.d/conda.sh \
      && conda config --set auto_update_conda false \
      && conda config --set show_channel_urls true \
      && conda config --set auto_activate_base false \
      && conda config --prepend channels conda-forge \
      && conda config --set channel_priority strict \
      && conda init bash

Step 3️⃣: After running this command, you will see a message in the terminal that ends with the statement For changes to take effect, close and re-open your current shell. Then, type

source ~/.bashrc

After that, the conda command will be available.

⁉️ FAQ: How do we set the Slow/Fast decomposition on MCMC chains with Cosmolike? Manual Blocking

The Cosmolike Weak Lensing pipelines contain parameters with different speed hierarchies. For example, Cosmolike execution time is reduced by approximately 50% when fixing the cosmological parameters. When varying only multiplicative shear calibration, Cosmolike execution time is reduced by two orders of magnitude.

Cobaya cannot automatically handle parameters associated with the same likelihood with different speed hierarchies. Luckily, we can manually impose the speed hierarchy in Cobaya using the blocking: option. The only drawback of this method is that parameters of all adopted likelihoods, not only the ones required by Cosmolike, must be manually specified.

In addition to that, Cosmolike can't cache the intermediate products of the last two evaluations, which is necessary to exploit optimizations associated with dragging (drag: True). However, Cosmolike caches the intermediate products of the previous evaluation, thereby enabling the user to take advantage of the slow/fast decomposition of parameters in Cobaya's main MCMC sampler.

Below, we provide an example YAML configuration for an MCMC chain with DES 3x2pt likelihood.

    likelihood: 
        des_y3.des_3x2pt:
        path: ./external_modules/data/des_y3
        data_file: DES_Y1.dataset
     
     (...)
     
    sampler:
        mcmc:
            covmat: "./projects/des_y3/EXAMPLE_MCMC22.covmat"
            covmat_params:
            # ---------------------------------------------------------------------
            # Proposal covariance matrix learning
            # ---------------------------------------------------------------------
            learn_proposal: True
            learn_proposal_Rminus1_min: 0.03
            learn_proposal_Rminus1_max: 100.
            learn_proposal_Rminus1_max_early: 200.
            # ---------------------------------------------------------------------
            # Convergence criteria
            # ---------------------------------------------------------------------
            # Maximum number of posterior evaluations
            max_samples: .inf
            # Gelman-Rubin R-1 on means
            Rminus1_stop: 0.015
            # Gelman-Rubin R-1 on std deviations
            Rminus1_cl_stop: 0.17
            Rminus1_cl_level: 0.95
            # ---------------------------------------------------------------------
            # Exploiting Cosmolike speed hierarchy
            # ---------------------------------------------------------------------
            measure_speeds: False # We provide the approximate speeds in the blocking
            # drag = false. The drag sampler requires the intermediate products of the last
            # two evaluations to be cached. Cosmolike can only cache the last evaluation.
            drag: False
            oversample_power: 0.2
            oversample_thin: True
            blocking:
            - [1,
                [
                    As_1e9, H0, omegab, omegam, ns
                ]
              ]
            - [4,
                [
                    DES_DZ_S1, DES_DZ_S2, DES_DZ_S3, DES_DZ_S4, DES_A1_1, DES_A1_2,
                    DES_B1_1, DES_B1_2, DES_B1_3, DES_B1_4, DES_B1_5,
                    DES_DZ_L1, DES_DZ_L2, DES_DZ_L3, DES_DZ_L4, DES_DZ_L5
                ]
              ]
            - [25,
                [
                    DES_M1, DES_M2, DES_M3, DES_M4, DES_PM1, DES_PM2, DES_PM3, DES_PM4, DES_PM5
                ]
              ]
            # ---------------------------------------------------------------------
            max_tries: 100000
            burn_in: 0
            Rminus1_single_split: 4

⁉️ FAQ: How do users set the environment for Machine Learning projects?

Commenting out the environmental flags shown below, located at set_installation_options script, will enable the installation of machine-learning-related libraries via pip.

[Adapted from Cocoa/set_installation_options.sh shell script] 
 
# ------------------------------------------------------------------------------
# If not set, Cocoa/installation_scripts/setup_pip_core_packages.sh will install several ML packages
# ------------------------------------------------------------------------------
#export IGNORE_EMULATOR_CPU_PIP_PACKAGES=1
#export IGNORE_EMULATOR_GPU_PIP_PACKAGES=1

We recommend using the GPU versions to train the emulator while using the CPU versions to run the MCMCs. This can be achieved by creating two separate environments, i.e., two independent CoCoA copies. Alternatively, users can manually incorporate the pip ML installation commands located on the shell script Cocoa/installation_scripts/setup_pip_core_packages.sh in their conda environments (two conda environments, one for the GPU and one for the CPU runs).

[Adapted from Cocoa/installation_scripts/setup_pip_core_packages.sh script - the command may not be entirely up to date] 
 
#CPU VERSION
env CXX="${CXX_COMPILER:?}" CC="${C_COMPILER:?}" ${PIP3:?} install \
    'tensorflow-cpu==2.12.0' \
    'tensorflow_probability-0.21.0' \
    'keras==2.12.0' \
    'keras-preprocessing==1.1.2' \
    'torch==1.13.1+cpu' \
    'torchvision==0.14.1+cpu' \
    'torchaudio==0.13.1' \
    'tensiometer==0.1.2' \
  --extra-index-url "https://download.pytorch.org/whl/cpu" \
  --prefix="${ROOTDIR:?}/.local" 

(...)

#GPU VERSION
env CXX="${CXX_COMPILER:?}" CC="${C_COMPILER:?}" ${PIP3:?} install \
    'tensorflow==2.12.0' \
    'tensorflow_probability-0.21.0' \
    'keras==2.12.0' \
    'keras-preprocessing==1.1.2' \
    'torch==1.13.1+cu116' \
    'torchvision==0.14.1+cu116' \
    'torchaudio==0.13.1' \
    'tensiometer==0.1.2' \
  --extra-index-url "https://download.pytorch.org/whl/cu116" \
  --prefix="${ROOTDIR:?}/.local"

⁉️ FAQ: How can users improve their Bash/C/C++ knowledge to develop Cocoa/Cosmolike? 📖📖

A working knowledge of Python is required to understand the Cobaya framework at the developer level. Users must also know the Bash language to understand Cocoa's scripts. Proficiency in C and C++ is also needed to manipulate Cosmolike and the C++ Cobaya-Cosmolike C++ interface. Finally, users need to understand the Fortran-2003 language to modify CAMB.

Learning all these languages can be overwhelming, so to enable new users to do research that demands modifications on the inner workings of these codes, we include here a link to approximately 600 slides that provide an overview of Bash (slides ~1-137), C (slides ~138-371), and C++ (slides ~372-599). In the future, we aim to add lectures about Python and Fortran.

⚠️⚠️ Warning about Weak Lensing YAML files in Cobaya

The CosmoLike pipeline takes $\Omega_m$ and $\Omega_b$, but the CAMB Boltzmann code only accepts $\Omega_c h^2$ and $\Omega_b h^2$ in Cobaya. Therefore, there are two ways of creating YAML compatible with CAMB and Cosmolike:

  1. CMB parameterization: $\big(\Omega_c h^2,\Omega_b h^2\big)$ as primary MCMC parameters and $\big(\Omega_m,\Omega_b\big)$ as derived quantities.

     omegabh2:
         prior:
             min: 0.01
             max: 0.04
         ref:
             dist: norm
             loc: 0.022383
             scale: 0.005
         proposal: 0.005
         latex: \Omega_\mathrm{b} h^2
     omegach2:
         prior:
             min: 0.06
             max: 0.2
         ref:
             dist: norm
             loc: 0.12011
             scale: 0.03
         proposal: 0.03
         latex: \Omega_\mathrm{c} h^2
     mnu:
         value: 0.06
         latex: m_\\nu
     omegam:
         latex: \Omega_\mathrm{m}
     omegamh2:
         derived: 'lambda omegam, H0: omegam*(H0/100)**2'
         latex: \Omega_\mathrm{m} h^2
     omegab:
         derived: 'lambda omegabh2, H0: omegabh2/((H0/100)**2)'
         latex: \Omega_\mathrm{b}
     omegac:
         derived: 'lambda omegach2, H0: omegach2/((H0/100)**2)'
         latex: \Omega_\mathrm{c}
    
  2. Weak Lensing parameterization: $\big(\Omega_m,\Omega_b\big)$ as primary MCMC parameters and $\big(\Omega_c h^2, \Omega_b h^2\big)$ as derived quantities.

Adopting $\big(\Omega_m,\Omega_b\big)$ as main MCMC parameters can create a silent bug in Cobaya; we are unsure if this problem persists in newer Cobaya versions, so this report should be a warning. The problem occurs when the option drop: true is absent in $\big(\Omega_m,\Omega_b\big)$ parameters, and there are no expressions that define the derived $\big(\Omega_c h^2, \Omega_b h^2\big)$ quantities. The bug is silent because the MCMC runs without any warnings, but the CAMB Boltzmann code does not update the cosmological parameters at every MCMC iteration. As a result, the resulting posteriors are flawed, but they may seem reasonable to those unfamiliar with the issue. Please be aware of this bug to avoid any potential inaccuracies in the results.

The correct way to create YAML files with $\big(\Omega_m,\Omega_b\big)$ as primary MCMC parameters is exemplified below

    omegab:
        prior:
            min: 0.03
            max: 0.07
        ref:
            dist: norm
            loc: 0.0495
            scale: 0.004
        proposal: 0.004
        latex: \Omega_\mathrm{b}
        drop: true
    omegam:
        prior:
            min: 0.1
            max: 0.9
        ref:
            dist: norm
            loc: 0.316
            scale: 0.02
        proposal: 0.02
        latex: \Omega_\mathrm{m}
        drop: true
    mnu:
        value: 0.06
        latex: m_\\nu
    omegabh2:
        value: 'lambda omegab, H0: omegab*(H0/100)**2'
        latex: \Omega_\mathrm{b} h^2
    omegach2:
        value: 'lambda omegam, omegab, mnu, H0: (omegam-omegab)*(H0/100)**2-(mnu*(3.046/3)**0.75)/94.0708'
        latex: \Omega_\mathrm{c} h^2
    omegamh2:
        derived: 'lambda omegam, H0: omegam*(H0/100)**2'
        latex: \Omega_\mathrm{m} h^2

💀 ☠️ 🛑👎 FAQ: How do we install Cocoa without conda (not recommended)

This method is slow and not advisable 🛑👎. When Conda is unavailable, the user can still perform a local semi-autonomous installation on Linux based on a few scripts we implemented. We require the pre-installation of the following packages:

To perform the local semi-autonomous installation, users must modify flags written on the shell scripts set_installation_options.sh and installation_scripts/flags_manual_installation.sh because the default behavior corresponds to an installation via Conda. First, select the environmental key MANUAL_INSTALLATION as shown below:

[Adapted from Cocoa/set_installation_options.sh script] 

# ------------------------------------------------------------------------------------
# HOW COCOA SHOULD BE INSTALLED? -----------------------------------------------------
# ------------------------------------------------------------------------------------
#export MINICONDA_INSTALLATION=1
export MANUAL_INSTALLATION=1

Finally, set the following environmental keys:

[Adapted from Cocoa/installation_scripts/flags_manual_installation.sh shell script]
# ------------------------------------------------------------------------------------
# IF SET, COCOA DOES NOT USE SYSTEM PIP PACKAGES -------------------------------------
# ------------------------------------------------------------------------------------
export DONT_USE_SYSTEM_PIP_PACKAGES=1

(...)

# ------------------------------------------------------------------------------------
# USER NEEDS TO SPECIFY THE FLAGS BELOW SO COCOA CAN FIND PYTHON / GCC ---------------
# ------------------------------------------------------------------------------------

export PYTHON_VERSION=XXX
export GLOBALPYTHON3=XXX
export GLOBAL_PACKAGES_LOCATION=XXX
export GLOBALPIP3=XXX
export GIT=XXX
export WGET=XXX
export CURL=XXX

# ------------------------------------------------------------------------------------

export PATH=XXX:$PATH
export CFLAGS="${CFLAGS} -IXXX"
export LDFLAGS="${LDFLAGS} -LXXX"
export C_INCLUDE_PATH=XXX:$C_INCLUDE_PATH
export CPLUS_INCLUDE_PATH=XXX:$CPLUS_INCLUDE_PATH
export PYTHONPATH=XXX:$PYTHONPATH
export LD_RUN_PATH=XXX:$LD_RUN_PATH
export LIBRARY_PATH=XXX:$LIBRARY_PATH
export CMAKE_INCLUDE_PATH=XXX:$CMAKE_INCLUDE_PATH
export CMAKE_LIBRARY_PATH=XXX:$CMAKE_LIBRARY_PATH
export INCLUDE_PATH=XXX:$INCLUDE_PATH
export INCLUDE=XXX:$INCLUDE
export CPATH=XXX:$CPATH
export OBJC_INCLUDE_PATH=XXX:$OBJC_INCLUDE_PATH
export OBJC_PATH=XXX:$OBJC_PATH
            
# ------------------------------------------------------------------------------------
# COMPILER ---------------------------------------------------------------------------
# ------------------------------------------------------------------------------------
export C_COMPILER=
export CXX_COMPILER=
export FORTRAN_COMPILER=
export MPI_CC_COMPILER=
export MPI_CXX_COMPILER=    
export MPI_FORTRAN_COMPILER=

# ------------------------------------------------------------------------------------
# FINE-TUNNING OVER THE USE OF SYSTEM-WIDE PACKAGES ---------------------------------
# ------------------------------------------------------------------------------------
#export IGNORE_XZ_INSTALLATION=1
#export IGNORE_DISTUTILS_INSTALLATION=1
#export IGNORE_C_GSL_INSTALLATION=1
#export IGNORE_C_CFITSIO_INSTALLATION=1
#export IGNORE_C_FFTW_INSTALLATION=1
#export IGNORE_CPP_BOOST_INSTALLATION=1
#export IGNORE_CMAKE_INSTALLATION=1
#export IGNORE_OPENBLAS_INSTALLATION=1
#export IGNORE_FORTRAN_LAPACK_INSTALLATION=1
#export IGNORE_CPP_ARMA_INSTALLATION=1
#export IGNORE_HDF5_INSTALLATION=1
#export IGNORE_EXPAT_CORE_PACKAGE=1
#export IGNORE_PIP_CORE_PACKAGES=1

⁉️ FAQ: How do we push changes to the Cocoa main branch? A few git hacks

Until recently, Cocoa development was unstructured, and developers could push directly to the main branch. Small commits were not discouraged, and such flexible development rules will soon change. Shortly, we will protect the main branch by requiring every push to be reviewed by Cocoa's leading developers. We also want to reduce the number of commits from now on.

Our new philosophy establishes that a commit in the main branch should contain an atomic change that takes code from a working state to another working state with meaningful and well-tested improvements. So, developers should propose changes to the main branch in larger chunks (via squash commits), as shown below.

Important note: we strongly advise developers to use the up-to-date git given by Cocoa conda environment.

  • ⁉️ How to apply squash commits?

Step 1️⃣: create a development branch from the main branch. Do not call the development branch dev, it is reserved for work done by the primary Cocoa developers. Let's call this new branch xyzdev for concreteness (tip: the use of developers' initials helps make the branch name unique)

# Developers must input the command below on the main branch.
"${CONDA_PREFIX}"/bin/git switch -c xyzdev

If the xyzdev already exists, use git switch without the -c flag. Developers can also push their development branches to the server via the command.

# Developers must input the command below on the xyzdev branch.
"${CONDA_PREFIX}"/bin/git push -u origin xyzdev

Step 2️⃣: develop the proposed changes. We advise developers to commit frequently. In your branch, a commit does not need to be atomic, changing the code from one working state to another well-tested, meaningful working state. In your branch, you have absolute freedom.

Step 3️⃣: Once the developers created an atomic, meaningful, and well-tested improvement to Cocoa, the developer needs to merge any subsequent changes made in main while the developer has been working on the xyzdev branch.

# Developers must input the command below on the xyzdev branch.
"${CONDA_PREFIX}"/bin/git merge main

This step may create conflicts that must be addressed before step four.

Step 4️⃣: Once the developer has merged recent changes made on the main branch, the developer must push to the main branch the modifications made on the xyzdev branch by first squashing all your changes into a single commit as shown below

# Developers must input the command below on the xyzdev branch.
"${CONDA_PREFIX}"/bin/git switch main

and

# Developers must input the command below on the main branch.
"${CONDA_PREFIX}"/bin/git merge --squash xyzdev

and

# Developers must input the command below on the main branch.
"${CONDA_PREFIX}"/bin/git commit -m "squash merge - xyzdev branch: added development on abc features"

and

# Developers must input the command below on the main branch.
"${CONDA_PREFIX}"/bin/git push origin main

Important note: never revert the branch ordering on squash merging by squashing the main changes to the xyzdev branch.

⁉️ FAQ: How do we develop from a git tag? A few more git hacks

A useful git hack is related to developing Cocoa from a git tag. We reserve git tags to set milestones in our development, so they are good starting points for coding localized new features (e.g., changes to a file that other developers have not recently modified) or bug fixes.

Step 1️⃣ (optional) If the developer has cloned the repository using the https URL address, we change the URL to the SSH-key-based address

"${CONDA_PREFIX}"/bin/git remote set-url origin [email protected]:CosmoLike/cocoa.git

Step 2️⃣ Move the detached state to a new local branch via the command

"${CONDA_PREFIX}"/bin/git switch -c mylocalbranch

Now, all commits will be associated with this local branch. The developer can then make a series of git commits to implement a new feature or fix a bug.

Step 3️⃣ The developer has two options at the end of development. They can either create a new remote branch named mylocalbranch

# Developers must input the command below on the mylocalbranch branch.
"${CONDA_PREFIX}"/bin/git push origin mylocalbranch

or they can fetch and download the remote branch, named myremotebranch, that will hold the changes made on mylocalbranch

# Developers must input the command below on the mylocalbranch branch.
"${CONDA_PREFIX}"/bin/git switch -c myremotebranch origin/myremotebranch

This will switch the repository to the myremotebranch. Now, the developer needs to merge the changes made on mylocalbranch. If mylocalbranch is NOT the main branch, type

# Developers must input the command below on the myremotebranch != main branch.
"${CONDA_PREFIX}"/bin/git merge mylocalbranch

If the developer wants to merge their changes to the mylocalbranch = main branch instead, do a squash merge

# Developers must input the command below on the main branch.
"${CONDA_PREFIX}"/bin/git merge --squash mylocalbranch

If this does not create any merge conflicts, type

# Developers must input the command below on the myremotebranch branch.
"${CONDA_PREFIX}"/bin/git push origin myremotebranch

⁉️ FAQ: How do we download and run Cosmolike projects?

The projects folder was designed to include Cosmolike projects. We assume that you are still in the Conda cocoa environment from the previous conda activate cocoa command and that you are in the cocoa main folder cocoa/Cocoa,

Step 1️⃣: Go to the project folder (./cocoa/Cocoa/projects) and clone a Cosmolike project with the fictitious name XXX:

cd ./cocoa/Cocoa/projects
"${CONDA_PREFIX}/bin/git" clone [email protected]:CosmoLike/cocoa_XXX.git XXX

By convention, the Cosmolike Organization hosts a Cobaya-Cosmolike project named XXX at CosmoLike/cocoa_XXX. The cocoa_ prefix must be dropped when cloning the repository.

Step 2️⃣: Go back to the Cocoa main folder and activate the private Python environment

cd ../
source start_cocoa.sh

⚠️ The start_cocoa.sh script must be run after cloning the project repository.

Step 3️⃣: Compile the project, as shown below

source "${ROOTDIR:?}"/projects/XXX/scripts/compile_XXX

Step 4️⃣: Select the number of OpenMP cores and run a template YAML file

export OMP_PROC_BIND=close; export OMP_NUM_THREADS=4
mpirun -n 1 --oversubscribe --mca btl vader,tcp,self --bind-to core --rank-by core --map-by numa:pe=${OMP_NUM_THREADS} cobaya-run ./projects/XXX/EXAMPLE_EVALUATE1.yaml -f

⚠️ Never delete the lsst_y1 folder from the project folder without running stop_cocoa first; otherwise, Cocoa will have ill-defined soft links. Where the ill-defined soft links will be located? They will be at Cocoa/cobaya/cobaya/likelihoods/, Cocoa/external_modules/code/ and Cocoa/external_modules/data/. The script stop_cocoa deletes them. Why this behavior exists? The script start_cocoa creates symbolic links so cobaya can see the likelihood and data files. It also adds the Cobaya-Cosmolike interface of all projects to LD_LIBRARY_PATH and PYTHONPATH paths.