Mounting data storages on local station

From MetaCentrum
Jump to: navigation, search

MetaCentrum has large data storing capacity accessible to users through NFSv4 protocol. On the compute nodes, you can reach this space in directory /storage/. You can find available storages and your quotas there either during login to frontend in ssh banner or at MetaCentrum web pages.

You can also access this space from your PC. This page contains Linux tutorial how to do it. There is no free client for MS-Windows, just comercial NFS Maestro.

NFSv4 directories and servers

NFS4 server adresář - directory velikost - capacity zálohovací třída - back-up policy alternativní jména serverů v Perunovi - alternative name / poznámka - note /storage/brno1-cerit/ 1.8 PB 2
/storage/brno2/ 110 TB 2 accessible through storage-brno6 temporarily, path ../fsbrno2/home/$LOGNAME /storage/brno3-cerit/ 932 TB 2 /storage/brno4-cerit-hsm/ zrušeno - decommissioned data archived in /storage/brno1-cerit/ /storage/brno5-archive/ 5 387 TiB 3 /storage/brno6/ 262 TB 2 /storage/brno7-cerit/ ruší se - decommissioned 2 data archived in /storage/brno1-cerit/ /storage/brno8/ 88 TB 3 in past /storage/ostrava1/ /storage/brno9-ceitec/ 262 TB 3 - pro NCBR CEITEC /storage/brno10-ceitec-hsm/ 3 dedicated to NCBR CEITEC /storage/brno11-elixir/ 313 TB 2 dedicated to ELIXIR-CZ, /storage/brno12-cerit/ 3.4 PB 0, domovský adresář v nfs4/home/$USER /storage/budejovice1/ 44 TB 3 (storage-cb1|storage-cb2) /storage/jihlava1-cerit/ zrušeno - decommissioned data archived to /storage/brno4-cerit-hsm/fineus,, symlink /storage/jihlava1-cerit/ /storage/jihlava2-archive/ zrušeno - decommissioned /storage/du-cesnet/ 3 /storage/liberec3-tul/ 30 TiB /storage/plzen1/ 352 TB 2 (storage-eiger1|storage-eiger2|storage-eiger3) /storage/plzen2-archive/ zrušeno - decommissioned /storage/plzen3-kky/ 73 TiB 3 /storage/praha1/ 100 TB 3 storage-praha1(a|b) /storage/praha4-fzu/ 15 TB /storage/praha5-elixir/ 157 TB 3 /storage/pruhonice1-ibot/ 179 TB 3
Zálohovací třídy jsou popsány v / Back-up policy is described at: Politika_zálohování (Back-up policy). Výtah/summary:
  • třída 2 - záloha (pouze) formou časových řezů / class 2 - backup (only) in a form of time slices
  • třída 3 - data se záložní kopií / class 3 - data with a backup copy

What do you need

You need several things to make NFSv4 from server accessible on your Linux desktop:

  • empty directory as a mount point
  • Kerberos support installed
  • NFSv4 support in kernel
  • kerberos ticket enabling connection
  • client NFSv4 tools
  • kerberos ticket to access to data
  • to be a root on your machine
  • set the system time properly

Directory /storage

Shared space is necessary to be map to some directory. We recommend directory /storage because in this way it is set on MetaCentrum machines. So create empty directory by command:

mkdir -p /storage/brno1
mkdir -p /storage/brno2
mkdir -p /storage/brno3-cerit
mkdir -p /storage/plzen1

Kerberos support

Install Kerberos system support. Packages heimdal-clients and krb5-config in Ubuntu 8.10 / Debian 9, packages krb5 and krb5-client in OpenSuse 11.1. It is necessary to set file /etc/krb5.conf properly. The easiest way is to copy it from some MetaCentrum machine e.g.machine

$ scp /etc

You can get kerberos ticket throught command kinit and to print it throught command klist on condition you have proper installation of Kerberos. If you are using MIT Kerberos >1.4 or Heimdal >1.3 add following line to the [libdefaults] section in the krb5.conf: allow_weak_crypto = true.

Support NFSv4 in kernel

Support is in the standard kernel in OpenSuse and Ubuntu so you don't have to test it. Otherwise assure the support in the following way:

Test of support NFS file system

  • grep nfs4 /proc/filesystems

Right answer: nodev nfs4

In case of empty answer do as a root

$ modprobe nfs

and repeat the test. In case of negativ answer it is necessary to compile NFS (with NFSv4 support) into kernel.

Test RPCSECsupport

  • ls -d /proc/net/rpc/auth.rpcsec*

Right answer: proc/net/rpc/auth.rpcsec.context /proc/net/rpc/auth.rpcsec.init

In case of answer: ls: cannot access /proc/net/rpc/auth.rpcsec*: No such file or directory</tt> do as root

modprobe auth_rpcgss

and repeat the test. In case of negative answer it is necessary to compile CONFIG_SUNRPC_GSS to Linux kernel.

Automatic inserting of modules

If it is obvious that system supports NFSv4 including RPCSEC it is not necessary to insert modules manually - client tools insert NFS on their own.

Kerberos ticket enabling volume connection

We offer 3 possibilities according type of tickets:

  • Your machine has file /etc/krb5.keytab from MetaCentrum (or ÚVT)
  • Your machine does not have file /etc/krb5.keytab from MetaCentrum
    • You can use your user ticket
    • We release one-function file /etc/krb5.keytab for you

Your machine has file /etc/krb5.keytab from MetaCentrum

We will add ticket like nfs/your_machine@ICS.MUNI.CZ to your /etc/krb5.keytab on your request which enables you to connect NFS volume. Original krb5.keytab must be changed to the new one, which will be released to you on your request.

Your machine does not have file /etc/krb5.keytab from MetaCentrum

You can use your own user ticket

Create kerberos ticket (kinit) before mounting the volume. You must create the ticket as a root because system volume will be connected as a root too. In this way, it is necessary to set rpc.gssd to search your ticket, not the system one, see bellow. Ticket must be renewed every day.

Create one-purpose file /etc/krb5.keytab

Login to some MetaCentrum machine. Be sure that you have valid Kerberos ticket in META realm (command klist will print it). After it use the command:

/software/remctl-2.12/bin/remctl -d accounts nfskeytab >krb5.keytab

File krb5.keytab will be created in actuall directory which contains one-purpose ticket like nfs/your_login@META. Contain of this file can be printed by command ktutil:

$ ktutil -k krb5.keytab list
Vno  Type         Principal
  2  des-cbc-crc  nfs/makub@META
  2  des-cbc-md4  nfs/makub@META
  2  des-cbc-md5  nfs/makub@META

Copy the file krb5.keytab on your machine to /etc/krb5.keytab, set its ACL to root.root and mode 600 as follows:

chown root.root /etc/krb5.keytab
chmod 600 /etc/krb5.keytab

Delete the file from MetaCentrum machine after copying it. It is important to be META your implicit realm in file /etc/krb5.conf.

Client NFSv4 tools

We recommend to install nfs-utils version 1.1.0 or higher.

  • It is package nfs-common (apt-get install nfs-common) on Debian/Ubuntu.
  • It is package nfs-utils (yast -i nfs-utils) on OpenSuse lower than 10.3
  • It is package nfs-client (yast -i nfs-client) on OpenSuse 10.3 and higher.

You will also need running portmap. It should be installed with nfs-utils dependencies. Otherwise it is package with the same name (portmap).

Setting of nfs-utils for NFSv4

  • Debian/Ubuntu

Setting of nfs-utils is in the file /etc/default/nfs-common. We set the values in the following way:

  • Opensuse

Setting of nfs-utils in the file /etc/sysconfig/nfs. We set at least following values:


/etc/fstab setting

Add the following lines to the /etc/fstab file /storage/brno1             nfs4       sec=krb5              0 0 /storage/brno2             nfs4       sec=krb5              0 0 /storage/brno3-cerit  nfs4       sec=krb5              0 0 /storage/plzen1           nfs4       sec=krb5              0 0

NOTE: For some combinations of operating systems or server/client, adding option vers=4 to fstab is necessary too: /storage/brno2             nfs4       sec=krb5,vers=4       0 0

Then start nfs and set auto start after reboot:


/etc/init.d/nfs start 
insserv /etc/init.d/nfs

Debian 9

systemctl restart
mount -a


In file /etc/default/nfs-common should be set the option NEED_GSSD=yes.

service rpcbind start
service nfs-common start
service rpc-gssd start
mount -a

For system older than 14.04:

service portmap start
service gssd start
mount -a

Simple settings of idmapd.conf

File /etc/idmapd.conf sets mapping of NFSv4 identities to local users (NFSv4 works with text principals of kerberos, POSIX interface of file system works with numeral representation of users and groups).

Simple settings /etc/idmapd.conf consist on setting of configuration line: Domain = META. The users from domain @META will be map throught files /etc/passwd and /etc/group. So it means that for identity xhejtman@META must exist record in /etc/passwd with name xhejtman. Ordinary tools (ls -l) will show the names properly if the name in given files will be exist for their principal. For nonexisting name will be user mapping choosen as nobody and nogroup.


grep xhejtman /etc/passwd
xhejtman:x:1000:1000:Lukas Hejtmanek,,,:/home/xhejtman:/bin/bash
ls -l /mnt/nfs/software
total 0
drwxr-xr-x 4 nobody   nogroup   51 2008-06-12 12:49 etics
-rw-r--r-- 1 nobody   nogroup    0 2008-06-06 14:26 hu
drwxr-xr-x 6 xhejtman soft-nfs4 54 2008-06-12 14:45 libnfsidmap
drwxr-xr-x 5 xhejtman soft-nfs4 40 2008-06-11 13:12 nsswitch

Particular record for user xhejtman@META exists in /etc/passwd and record for group soft-nfs4@META also exists in /etc/group, so mapping runs properly and it is also properly shown. There is no record for users who own directory etics that is why it's shown as nobody:nogroup.

Mounting the volume

You don't need to mount a volume in OpenSuse 11.1, because running nfs service connect it automatically according records in /etc/fstab, in other OS, explicit mounting (mount -a) is needed. If you followed above steps, you should be able to list /storage/* dirs now.

In other case, mounting the volume is necessary to do as a root. Be sure that you have proper /etc/krb5.keytab or user ticket created as a root and be sure that portmap and nfs-common (nfs-utils) are running. You can assure yourself throught commands

ps ax | grep rpc.gssd, 
ps ax | grep portmap.

We make connection throught:

mount -t nfs4 /storage/brno1 -o sec=krb5i,vers=4.0
mount -t nfs4 /storage/brno2 -o sec=krb5i,vers=4.0
mount -t nfs4 /storage/plzen1 -o sec=krb5i,vers=4.0

Names of NFSv4 servers are described above in this page.

You can insert any other local directory where we want to mount NFS volume instead of /storage. If you use choice sec=krb5i instead of sec=krb5, data integrity will be check during transfer. If you use choice sec=krb5i data will be transfered in code.

Accessing user data on NFS4 storage

If you use the tutorial you should have the volume mounted now. To be able to access your user data you must have valid kerberos ticket. You can obtain one by command kinit. So:

$ kinit
login@META's Password:
$ cd /storage/brno2/home/login

Please keep in mind that such kerberos ticket has limited validity (usually 12h). If you need to access storage without entering password and/or for longer periods (i.e. running some service), you can create ticket from one-purpose nfs/login@META keytab (see above how to obtain it):

kinit -t /etc/krb5.keytab nfs/login@META

With such a ticket you should be able to access login's data, however, it has limited (12h) validity too. Please, prolong it as necessary:

echo "* */6 * * * /usr/bin/kinit -t /etc/krb5.keytab  nfs/login@META" | crontab

In case of problems, please write to our user support

Kerberos identity

Users using one identity should be satisfied with standart NFS tools in their Linux distribution. If you want to use more identities at one time like xhejtman@META, xhejtman@ADMIN.META and xhejtman@ICS.MUNI.CZ it's necessary to use patched rpc.gssd program. Patched program is in NFS utils version 1.1.3. Patching older version is not easy, best way is to contact us and we make patched package for you.

Kerberos tickets

Ticket in /etc/krb5.keytab is implicitly used just to mount the volume. But we can make a deal that the ticket can be also used to access the storage – but explicit arrangement is necessary. This ticket is valid until kerberos server administrator deletes it. So the access via ticket can be used almost forever.

User ticket (ussually in /tmp/krb5cc_number) can be used for mounting (if we run rpc.gssd with choice -n) and also for access. This ticket often has limited validity.

Settings of proper displaing of users and groups names

Above-mentioned settings of idmapd.conf will display properly only users and groups stored in /etc/passwd and /etc/group. Moreover the user or group must be from META domain. Cross realm user mapping is possible through advanced settings. It is necessary to set mapping of NFSv4 identities to numeral representation and it is necessary to set mapping of numerical representation to the individual names.

Settings of cross realm mapping NFSv4 identities to numerical representation

It is necessary to reach mapping files of users and groups for mapping support.

You have to save this files to /etc/passwd-nfs4 and /etc/group-nfs4.

Then you have to download and install new version of library libnfsidmap from NFS. It is placed in /storage/software/libnfsidmap, where are both libraries for IA32 and X86_64 (lib32 and lib64), and Debian packages (for i386 a ndAMD64).

For this way of mapping it is necessary to set simply way, desribed above, get a packages and then you can try this advanced way. We should change settings of idmapd.conf after installing new version and reaching mapping files. :

Method = mnsswitch

Similar settings could have been in configuration file. New settings have text: mnsswitch instead nsswitch. We keep settings Domain = META the same.

Restart service idmapd.

Debian: /etc/init.d/nfs-common restart, 
opensuse /etc/init.d/idmapd restart or /etc/init.d/nfs restart

So now we have cross realm mapping of identities NFSv4 to numeral representation for POSIX interface.

Settings of cross realm mapping of numeral representation to the names

It is settings of nsswitch mechanism of name translation.

You should download library from NFS /storage/software/nsswitch and place it into directory /lib. There are versions for debian i386 a X86_64 at NFS. There is no packages because it is only single file.

Change configuration of /etc/nsswitch.conf in following way:

passwd:         compat nfs4
group:          compat nfs4

Add word nfs4 to the end of the lines passwd and group (it is the same name as have library libnss-jmé

You don't have to restart service, new mapping should work immediatelly.

Names with domains are written implicitly. If we don't want to write some domain again and again, it is possible to export environment variable NFS4DOMAIN=META.

Example: export NFS4DOMAIN=META, 

then names from domain META will be shown without this domain in the list.

Installation way for a Gentoo

You need to install packages net-nds/portmap and net-fs/nfs-utils. Check whether nfs-utils are compilled with kerberos.

Scripts for a start nfs in Gentoo are little bit odd because they have common settings of server and client. Settings of nfs-utils is in the file /etc/conf.d/nfs where you need to change just choice "OPTS_RPC_GSSD" to the value " -- -n " in case you use your own key and not machine keytab.

Then assure you have downloaded modules nfs a rpcsec_gss_krb5 and set them to load aftear booting the system. The scripts can't load them itself.

Configure /etc/idmapd.conf, as is described upper, simply just rewrite Domain to META in pattern file and return to this place in the tutorial.

You need run services rpc.gssd and rpc.idmapd, for example /etc/init.d/rpc.gssd start, and manage to be run after start (rc-add default rpc.gssd and similary for rpc.idmapd).

Add into /etc/fstab line /storage nfs4 sec=krb5

and run service /etc/init.d/nfsmount. Volume should mount now.