Accessing the Software

Print

 

* Compute Canada documentation: https://docs.computecanada.ca/wiki/Utiliser_des_modules/en

* Using Compute Canada software stack on Guillimin: https://wiki.calculquebec.ca/w/Ex%C3%A9cuter_une_t%C3%A2che/en#tab4

Using software modules

The access to different software packages, installed on Guillimin, is easily managed by users via the "Lmod" environment (https://www.tacc.utexas.edu/research-development/tacc-projects/lmod). It automatically adds/removes corresponding paths to your environment variables, thus removing the necessity to do it "by hand". Note: as of March 22, 2016, a new, mostly compatible, modules environment, Lmod replaced the old modules environment (http://modules.sourceforge.net/). Most users see no differences as existing (legacy) modules are retained. There are however three visible differences:

  1. A hierarchical toolchain with a new set of modules can be accessed by doing either:
    1. module load iomkl/2015b (loading Intel Compiler + OpenMPI + MKL), or
    2. module load foss/2015b (loading GCC 4.9.3 + OpenMPI + OpenBLAS)

    After this, “module avail” shows a longer list of modules with newer software versions.

  2. It is no longer possible to load two versions of the same module at the same time. If you try to load both “gcc/4.8.2” and “gcc/4.9.1” the latter completely replaces the former.
  3. Caching: if you maintain your own modulefiles please be aware that they are cached:
    1. You can force regeneration by doing:
      user@guillimin$ rm -rf ~/.lmod.d/.cache
      
    2. You can disable caching by doing:
      user@guillimin$ export LMOD_SHORT_TIME=86400
      

The following commands of the "Modules" environment are used most frequently:

  • Show all modules available on the system:
    user@guillimin$ module avail
    
  • Show currently loaded modules:
    user@guillimin$ module list
    
  • Load particular module:
    user@guillimin$ module add <modulename>
    
  • Unload particular module:
    user@guillimin$ module rm <modulename>
    

PLEASE NOTE:  To be completely sure you have unloaded a particular module it is best practice to logout of the system and re-login.  That will guarantee the environment contains no references to a specific module.

New Lmod/EasyBuild-based module system

As the old module system has several issues that can lead to inconsistencies in the software environment, and also is showing its age, we have implemented a new module system in parallel based on Lmod and EasyBuild.

Easybuild defines toolchain modules to organize compilers, MPI implementations, and numerical libraries.

There exist 3 levels:

  1. Compiler only:
    1. GNU: GCC + updated binutils
    2. iccifort: Intel compilers + GNU
    3. pgi: PGI compilers + GNU
  2. Compiler + MPI
    1. gompi: GNU + OpenMPI
    2. gmvapich2: GNU + MVAPICH2
    3. iompi: Intel + OpenMPI
    4. imvapich2: Intel + MVAPICH2
    5. pompi: PGI + OpenMPI
    6. pmvapich2: PGI + MVAPICH2
    7. iimpi: Intel + Intel MPI
  3. Compiler + MPI + numerical libraries (BLAS, LAPACK, SCALAPACK, FFTW)
    1. iomkl/2015b (recommended): iompi + Intel MKL
    2. foss/2015b: Free and Open Source Software: gompi + OpenBLAS + FFTW + SCALAPACK.
    3. intel/2015b: iimpi + Intel MKL

It is available as follows:

  • Option 1: leaving all old modules in the "module avail" output  (the default as of March 22, 2016):
    • Using “module avail” you will see a list of all modules plus the new toolchain modules
    • You can use “module load iomkl/2015b” to see many new modules and have:
      • gcc = GCC 4.9.3
      • icc = Intel compiler 2015.3
      • mpicc = OpenMPI 1.8.8 + icc
    • Use “module switch iomkl foss” to switch icc+MKL to gcc+OpenBLAS.
  • Or option 2: having only the new modules visible by default:
    • create a file named ~/.lmod:
      user@guillimin$ touch ~/.lmod
    • This then loads the iomkl/2015b module by default (similar to Colosse):
      • gcc = GCC 4.9.3
      • icc = Intel compiler 2015.3
      • mpicc = OpenMPI 1.8.8 + icc
    • Use “module load legacy” to access the old modules.
    • Use “module switch iomkl foss” to switch icc+MKL to gcc+OpenBLAS.

Most old modules have equivalents now. Some major examples:

  • Scripting: Python, Perl, Octave, Ruby, R
  • Molecular Dyn., Electronic Structure: GROMACS, NWChem, NAMD, LAMMPS, Quantum Espresso, CP2K
  • High Energy Physics: CLHEP, Geant4
  • Libraries: HDF5, NetCDF, GSL, PETSc
  • CFD: OpenFOAM
  • Visualization: gnuplot, grace, Paraview
  • Programming: CUDA, Clang, Java

The most important new Lmod command is module spider

Example:

user@guillimin$ module spider openfoam
  OpenFOAM:
     Versions:
        OpenFOAM/1.7.1-GCC-OpenMPI
        OpenFOAM/2.1-GCC-OpenMPI
        OpenFOAM/2.1-GCC-4.8.2-OpenMPI
        ....
        OpenFOAM/2.3.1
-----------------------------------------------------------------
  To find detailed information about OpenFOAM please enter the full name.
  For example:
     $ module spider OpenFOAM/2.3.1

user@guillimin$  module spider OpenFOAM/2.3.1
-----------------------------------------------------------------
  OpenFOAM: OpenFOAM/2.3.1
-----------------------------------------------------------------
        Description:
      OpenFOAM is a free, open source CFD software package. OpenFOAM has an extensive range of features to solve anything from complex fluid flows
      involving chemical reactions, turbulence and heat transfer, to solid dynamics and electromagnetics. - Homepage: http://www.openfoam.com/ 

     Other possible modules matches:
        OpenFOAM-Extend

    You will need to load all module(s) on any one of the lines below before the "OpenFOAM/2.3.1" module is available to load.

      foss/2015b
      iomkl/2015b

    Help:
      OpenFOAM is a free, open source CFD software package.
       OpenFOAM has an extensive range of features to solve anything from complex fluid flows
       involving chemical reactions, turbulence and heat transfer,
       to solid dynamics and electromagnetics. - Homepage: http://www.openfoam.com/

The command “module switch” even works to change the toolchain for already loaded modules:

user@guillimin$ module load iomkl/2015b OpenFOAM/2.3.1
user@guillimin$ module switch iomkl/2015b foss/2015b
Due to MODULEPATH changes the following have been reloaded:
  1) OpenFOAM/2.3.1  2) OpenMPI/1.8.8  3) SCOTCH/6.0.4