From MetaCentrum
Jump to navigation Jump to search
Singularity logo.svg

Singularity is a free, cross-platform and open-source computer program that performs operating-system-level virtualization also known as containerization.

Singularity is able to support natively high-performance interconnects, such as InfiniBand[24] and Intel Omni-Path Architecture (OPA). It also has native support for Open MPI library by utilizing a hybrid MPI container approach where OpenMPI exists both inside and outside the container. Singularity can import Docker images without having Docker installed or being a superuser.

On the contrary to Docker, Singularity was designed do fit the high-performance computing (HPC) needs. HPC environments are typically multi-user systems where users should only have access to their own data. For all practical purposes, Docker gives superuser privileges. It’s hard to give someone limited Docker access. Singularity, on the other hand, runs under user identity. It blocks privilege escalation inside containers by using an immutable single-file container format that can be cryptographically signed and verified.


Singularity is installed on all MetaCentrum and Cerit nodes. You can also try experimental version from development branch available in /opt/singularity.

Basic usecases

Some basic usecases covering the singularity usage are bellow. Please note, that mentioning all nuances (especially usage of various versions of MPI or running parallel job on different infiniband HW) is beyond scope of this section.

Interactive session

[dexter@ungu1 ~]$ singularity shell my_image.img
Singularity: Invoking an interactive shell within container...

Running command

[dexter@ungu1 ~]$ singularity exec my_image.img bash -c "java -version"
java version "1.8.0_60"
Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)

PBS Pro: singularity interactive session

qsub -I -l select=1 -l walltime=24:00:00 -- /usr/bin/singularity shell my_image.img

PBS Pro: running script inside singularity container

qsub -l select=1 -l walltime=24:00:00 -- /usr/bin/singularity exec -B /path/to/script:/home/username/ my_image.img bash -c "/home/username/"

The -B /path/to/script:/home/username/ option will bind the host directory (/path/to/script) to container directory (in this example /home/username). Without this option, the container will automatically bind to itself host directories on computational node where the job is run and the script may not be found.

PBS Pro: running parallel job using singularity

The scenario for this setup is: two nodes with common scratch dir

 #PBS -l select=2:ncpus=2:mem=1gb:scratch_shared=4gb
 #PBS -l walltime=04:00:00
 #PBS -l place=scatter
 # modify/delete the above given guidelines according to your job's needs

 module add openmpi-2.0.1-gcc
 cat $PBS_NODEFILE |uniq >nodes.txt

 # run job over ethernet or infiniband (mpirun autoselects better)
 mpirun -n 2 --hostfile nodes.txt singularity exec my_image.img /path/to/program

More information about parallelization and different setups (specially for programs supporting MPI and OpenMP together) can be found in Parallelization.

Preparing your own singularity image

Preparing your own singularity image is intended for experienced users. Root privileges may be needed. Reading singularity documentation singularity documentation is a good idea too :) In general, you do not need root privileges if you can (re)use existing docker image.

Without root privileges you can do simply:

singularity image.create -s size_in_mb image.img
singularity build image.img docker://tensorflow/tensorflow:latest

However, if you want to change something or make your own image from scratch, you'll need root privileges to be able to write (-w) into container image:

singularity image.create -s size_in_mb image.img
singularity build -w image.img docker://ubuntu:latest # build base image from Docker hub
singularity shell -w image.img
apt-get install my_software

Starting docker image

qsub -l select=1 -l walltime=24:00:00 -- /usr/bin/singularity exec
docker://ubuntu:latest echo "Hello Dinosaur!"

more details:

Starting application docker image

The docker download instructions of the type

docker pull sangerpathogens/circlator

are in singularity replaced as

singularity pull docker://sangerpathogens/circlator

It will create circlator.simg, singularity image of docker image. Then if you have commands to start image with mounted folders, eg. by

docker run -v /home/ubuntu/data:/data sangerpathogens/circlator

use singularity binding by

mkdir circ_read 
singularity run -B ./circ_read/:/data ./circlator.simg

where circ_read is folder used for getting data into image. By running the command you are in the image and you can check that the folder is already mounted by

df -h

To run script or command, eg. here circlator, in the image you can use

 singularity exec -B ./circ_read/:/data ./circlator.simg "circlator"

inside the quotes, there is command that will be run inside the image. If you are using binding of specific directory (mostly containing input and output data), use absolute paths to the inputs (eg. /data/some.fasta) that are used as command parameters. After the exec you are back in standard environment (outside the image), here you must such paths (eg. circ_read).

Environment Settings (optional)

Before you start Singularity you may need to set:

export SINGULARITY_CACHEDIR="/storage/..."
# Than you can start Singularity
singularity build ...
  • CACHEDIR -- downloaded layers
  • LOCALCACHEDIR -- run shell exec
  • TMPDIR -- squashfs and temporary files, there is limit 1GB by default, if you need more use scratch



Program administrator