Docker in Podman

Metacentrum wiki is deprecated after March 2023 Dear users, due to integration of Metacentrum into https://www.e-infra.cz/en (e-INFRA CZ service), the documentation for users will change format and site. The current wiki pages won't be updated after end of March 2023. They will, however, be kept for a few months for backwards reference. The new documentation resides at https://docs.metacentrum.cz.

There is a possibility to run Docker images, currently limited only to Cerit machines ( frontend zuphux, PBS server cerit-pbs.cerit-sc.cz). The issue of root privileges inherent to Docker is circumvent by using Podman instead of Docker.

In what follows, we give an example using ubuntu Docker image. For a different image, the batch script has to be modified in a relevant way.

Example: run Ubuntu Docker image

Suppose you want to run Docker image ubuntu from Docker Hub, start an ssh server on the system and have possibility to log in by ssh. Prepare batch script run.sh

#!/usr/bin/podmanwrapper ubuntu:18.04 
#PODMAN_OPT -p 2222:22/tcp
 
  # commands that follow are specific for this usecase
apt-get update # install and start ssh server on ubuntu 
export DEBIAN_FRONTEND=noninteractive # don't ask the user for e.g. timezone etc.
apt-get -y -o Dpkg::Options::="--force-confdef" -o DPkg::Options::="--force-confold" install openssh-server
apt-get -y install openssh-server
service ssh start 
mkdir /root/.ssh
  
  # in the next command, you need to substitute my_ssh_publickey by your SSH public key. 
  # If you have one, default location is ~/.ssh/id_rsa.pub
  # If you do not have the SSH key, you need to generate one (ssh-keygen command)
  #echo "my_ssh_publickey" > /root/.ssh/authorized_keys
  # e.g.
echo "ssh-rsa AAAAB3N...d0ET0iUOpu==" > /root/.ssh/authorized_keys
chmod 0755 /root/.ssh
chmod 0600 /root/.ssh/authorized_keys
 
  #make nvidia drivers accessible
cp /host/usr/lib/nvidia/current/nvidia-smi /usr/bin 
cp -rP /host/usr/lib/x86_64-linux-gnu/nvidia/current/* /usr/lib/x86_64-linux-gnu
cp -rP /host/usr/lib/mesa-diverted/x86_64-linux-gnu/* /usr/lib/x86_64-linux-gnu
 
  #start bash session
/bin/bash

Then submit the script like a normal batch job:

[melounova@zuphux ~]$ qsub -l select=mem=4gb:scratch_local=1gb:os=debian10 -l walltime=02:00:00 run.sh
337811.cerit-pbs.cerit-sc.cz

For now Podman can be used only on hosts with Debian 10 (os=debian10). From another front-end than zuphux you have to specify Cerit PBS server by -q @cerit-pbs.cerit-sc.cz.

Login to ubuntu image and link necessary directories

Find out where the job is running by qstat -f job_number command:

[melounova@zuphux ~]$ qstat -f 337811.cerit-pbs.cerit-sc.cz | grep host

exec_host = glados12/5
exec_host2 = glados12.cerit-sc.cz:15002/5

Alternatively you can find the host at PBSmon.

Login as root to the container running on a computational node. Don't forget to specify the port by using ssh -p port_number option.

[melounova@zuphux ~]$ ssh -p 2222 root@glados12.cerit-sc.cz

NOTE: If you work from different frontend than zuphux, you have to specify that the id_rsa from your home at brno3-cerit storage is used:

ssh -i /storage/brno3-cerit/home/melounova/.ssh/id_rsa -p 2222 root@zenon31.cerit-sc.cz

All host's filesystem is in directory /host (e.g. /host/mnt, /host/storage). Within this directory, you have very limited rights. You can access them by making links within your root directory.

# test if I am logged in as root
root@glados12:~# whoami 
root 

# root directory contain the generic Linux subdirectories plus /host directory
root@glados12:~# ls / 
bin  boot  dev  etc  home  host  lib  lib64  media  mnt  opt  proc  root  run  sbin  scratch  srv  sys  tmp  usr  var
root@glados12:~# ls /host
afs aquota.user boot dev home initrd.img lib lib64 lost+found mnt packages puppet6-release-buster.deb run scratch.ssd srv sys  usr  vmlinuz aquota.group bin cvmfs etc ics   initrd.img.old lib32 libx32 media opt proc root sbin  software storage tmp var vmlinuz.old

# in /host/mnt and /host/storages there are /storage directories, but they cannot be accessed directly and must be linked
root@glados12:~# ls /host/mnt 
storage-brno1-cerit        storage-brno12-cerit  ... storage-liberec3-tul  storage-praha1
root@glados12:~# ls /host/storage
brno1-cerit brno11-elixir  ... praha4-fzu  projects

# link all directories in /host/mnt/ to /mnt (alternatively use: rmdir /mnt ; ln -s /host/mnt /mnt)
root@glados12:~# ln -s /host/mnt/* /mnt/ 
root@glados12:~# ls /mnt
storage-brno1-cerit  storage-brno12-cerit  ...  storage-liberec3-tul  storage-praha1

# /do the same with /host/storage directory
root@glados12:~# ln -s /host/storage /storage 
root@glados12:~# ls /storage
brno1-cerit brno11-elixir ...  praha4-fzu  projects