Application modules old

From MetaCentrum
Jump to navigation Jump to search

The system of modules used in Metacentrum provides user environment necessary to run requested applications; it allows to configure the user environments independently on the server's platform or files' locations. To run a certain software, the user must first load a module for this software.

In simple words: module = application = software.

Note:When used on frontends, the modules may behave a little differently than on computational nodes. The reason is that frontends are intended for interactive user work (job preparation, manipulation of data etc.) and may have therefore different system settings than computational nodes. If you need to run environment-sensitive operation (typically highly optimized compilations), do not test it on frontend, but request an interactive job and run your test compilations on the same node on which your final compilation will be done.

Usage

The basic command to work with modules is module.

Get a list of all available modules (applications):

$ module available

Abbreviated forms of this command:

$ module avail
$ module av

Get e.g. list of all modules containing "Matlab":

$ module avail matlab # list all modules starting with "matlab"
$ module avail {M,m}atlab # list all modules starting with "matlab" or "Matlab"

Use grep to search through list of modules:

$ module avail 2>&1 | grep -i atlab # list all modules containing string "atlab" in their name
$ module avail 2>&1 | grep -i 'atlab-9\.' # list all modules containing string "atlab-9." in their name

Load a specified module:

$ module add  matlab-9.9  # load Matlab ver. 9.9, for example

Limit the module validity to a bash subshell

Within a single session you can use modules separately by enclosing them to parentheses. Parentheses, in general, create a subshell in bash, i.e. all modules loaded within a certain subshell will be unloaded automatically after the subshell exits.

Example: normally the following two modules will conflict if loaded within the same session:

(BUSTER)melounova@skirit:~$ module add python/3.8.0-gcc
This is the user wrapper module using spack module python/3.8.0-gcc-rab6t.
(BUSTER)melounova@skirit:~$ module add python/3.8.0-gcc-rab6t
python/3.8.0-gcc-rab6t(49):ERROR:150: Module 'python/3.8.0-gcc-rab6t' conflicts with the currently loaded module(s) 'python/3.8.0-gcc'
python/3.8.0-gcc-rab6t(49):ERROR:102: Tcl command execution failed: conflict python

However you can load them separately in subshells:

(BUSTER)melounova@skirit:~$ (module add python/3.8.0-gcc; python)
This is the user wrapper module using spack module python/3.8.0-gcc-rab6t.
...
>>> quit()
(BUSTER)melounova@skirit:~$ (module add python/3.8.0-gcc-rab6t; python) 
...
>>> quit()

List currently loaded modules:

$ module list
Currently Loaded Modulefiles:
  1) metabase     2) matlab-9.9

Unload a module:

$ module unload matlab-9.9      # unload matlab-9.9, for example
$ module del matlab-9.9         # equivalent to the above

Clear all currently loaded modules:

$ module purge              # get rid of everything!

Display information about the module:

$ module display tensorflow-2.0.0-gpu-python3

-------------------------------------------------------------------
/packages/run/modules-2.0/modulefiles/tensorflow-2.0.0-gpu-python3:

module		 add python36-modules-gcc # these modules will be loaded, too
module		 add cudnn-7.6.4-cuda10.1 
prepend-path	 PATH /software/tensorflow/2.0.0/python36-gpu/bin # path to where the software is installed
prepend-path	 PYTHONPATH /software/tensorflow/2.0.0/python36-gpu/lib/python3.6/site-packages # another path to some python libraries
-------------------------------------------------------------------

Module vs modulefile

"A module" consists of 2 parts: the binary (executable), which is the software itself, and a modulefile, which is a text file defining paths to the binary, paths to external libraries and possible dependencies on other modules.

In simple words: module = binary + modulefile.

The modulefiles reside in some of the directories included in variable MODULEPATH (try echo $MODULEPATH).

For many applications, there are several modules available. This means there are several copies of the software available, differing in version, compiler and other features. For example,

(BUSTER)melounova@skirit:/packages/run/modules-2.0/modulefiles$ ls tensorflow*
tensorflow-1.13.1-gpu-python3  tensorflow-1.3.0-gpu          tensorflow-1.5.0-cpu-python3  tensorflow-1.7.1-gpu-python3  
tensorflow-1.3.0-cpu           tensorflow-1.3.0-gpu-python3  tensorflow-1.5.0-gpu          tensorflow-2.0.0-gpu-python3
tensorflow-1.3.0-cpu-python3   tensorflow-1.4.1-gpu          tensorflow-1.7.1-cpu-python3

says there are 11 versions of TensorFlow software differing by version, Python version (2 or 3) and whether it can be used for GPU computing (gpu) or not (cpu).

Modulefile example

#%Module1.0
#!
#! Title: gsl-1.16
#! Platforms: amd64_linux32
#! Version: 1.16
#! Description: GNU Scientific Library tools collection
#!
#! Author: Petr Hanousek, #35289 
#!
#!
proc ModulesHelp {} {
global ModulesCurrentModulefile
puts stdout "modulehelp $ModulesCurrentModulefile"
}

set basedir /software/gsl/1.16/gcc

prepend-path PATH ${basedir}/bin 
prepend-path PKG_CONFIG_PATH ${basedir}/lib/pkgconfig
prepend-path LD_LIBRARY_PATH ${basedir}/lib
prepend-path MANPATH ${basedir}/share/man
  • the number after the author's name you can see in some modulefiles is the number of RT ticket which requested the installation
  • the only necessary path is PATH, where the executable is
  • a modulefile can add other modules (module add ...) if there is a dependency
  • basedir is path to the install directory - it can be pretty anywhere; in case of non-standard installation, e.g. to home directory, don't forget to set variable MODULEPATH accordingly (see How to install an application)

Generic module

In some cases (typically when the only difference is in version) there is a generic module without the version name, which is nothing more than an alias of a newest version.

For example, module matlab is (currently) a shortcut for matlab-9.9.

$ module display matlab
-------------------------------------------------------------------
/packages/run/modules-2.0/modulefiles/matlab:
prepend-path	 PATH /software/matlab-9.9/bin:/software/matlab-meta_ext/bin 
prepend-path	 MATLABPATH /software/matlab-meta_ext/ 
-------------------------------------------------------------------

Documentation