File systems in MetaCentrum

Z MetaCentrum
Přejít na: navigace, hledání
Related topics
Working with data

Deciding what type of storage to use for your data affects how efficiently you are able to manage your job output. Criteria you should consider include

  • how much data you need to store
  • how long you intend to store the data
  • how easily you want to be able to access data
  • how quickly you need to transfer data

File systems available

The disk space in MetaCentrum must cater for various needs, so there are several types of if:

All the computing nodes provide three types of storage volumes, which are available for users:

  • /scratch - fast filesystem, dedicated for applications' temporal/working data. This volume is local on every computing node, provides a basic protection against HW failures (RAID 10) and is not backed up.
  • /home - shared filesystem dedicated for users' homes.
  • /storage - MetaCentrum's shared and backed-up NFSv4 volumes (available from the frontend as well as all the computing nodes)


Temporary scratch storage (for temporary data)

The scratch volumes are very fast filesystems intended to be used as a temporary storage for running jobs. The filesystems are located either on a local discs of a node (data cannot be shared among nodes) or on a fast network storage (data can be shared among the nodes of a cluster) -- the $SCRATCHDIR environment variable always points to a scratch space dedicated to a job. These volumes should be used only by currently running jobs, so any data must be removed immediately after a job finishes.


/storage volumes NFS

/storage volumes are built on the NFSv4 network filesystem. Physical storages are disk arrays situated in all cities where are clusters connected to MetaCentrum.

These volumes serves for storing users data mainly to folders /storage/city/home/user. They are available at machines with "nfs4" property.

Kerberos authentication system is used to manage the user access and you can mount this filesystem from every local machine.

Unfortunately the speed of data writing operation is influenced by the length of data network. Therefore writing to /storage/brno{1.2.3} located in Brno will be the fastest from the Brno's clusters then from Prague clusters and the slowest from Plzen clusters. The influence is worse while writing small files than for larger ones. The reading speed is not much affected.

Zálohovací třídy jsou popsány v / Back-up policy is described at: Politika_zálohování (Back-up policy). Výtah/summary:

  • třída 2 - záloha (pouze) formou časových řezů / class 2 - backup (only) in a form of time slices
  • třída 3 - data se záložní kopií / class 3 - data with a backup copy
NFS4 server adresář - directory velikost - capacity zálohovací třída - back-up policy alternativní jména serverů v Perunovi - alternative name / poznámka - note
storage-brno1-cerit.metacentrum.cz /storage/brno1-cerit/ 1.8 PB 2 nfs-ntc.ics.muni.cz
storage-brno2.metacentrum.cz /storage/brno2/ 110 TB 3 (nienna1|nienna2|nienna-home).ics.muni.cz
storage-brno3-cerit.metacentrum.cz /storage/brno3-cerit/ 932 TB 2 nfs-kat.cerit-sc.cz
storage-brno4-cerit-hsm.metacentrum.cz /storage/brno4-cerit-hsm/ zrušeno - decommissioned data archived in /storage/brno1-cerit/
storage-brno5-archive.metacentrum.cz /storage/brno5-archive/ 5 387 TiB 3 nfs.du3.cesnet.cz
storage-brno6.metacentrum.cz /storage/brno6/ 262 TB 2
storage-brno7-cerit.metacentrum.cz /storage/brno7-cerit/ ruší se - decommissioned 2 data archived in /storage/brno1-cerit/
storage-brno8.metacentrum.cz /storage/brno8/ 88 TB 3 in past /storage/ostrava1/
storage-brno9-ceitec.metacentrum.cz /storage/brno9-ceitec/ 262 TB 3 storage-ceitec1.ncbr.muni.cz - pro NCBR CEITEC
storage-brno10-ceitec-hsm.metacentrum.cz /storage/brno10-ceitec-hsm/ 3 dedicated to NCBR CEITEC
storage-brno11-elixir.metacentrum.cz /storage/brno11-elixir/ 313 TB 2 dedicated to ELIXIR-CZ
storage-budejovice1.metacentrum.cz /storage/budejovice1/ 44 TB 3 (storage-cb1|storage-cb2).metacentrum.cz
storage-jihlava1-cerit.metacentrum.cz /storage/jihlava1-cerit/ zrušeno - decommissioned data archived to /storage/brno4-cerit-hsm/fineus, storage-brno4-cerit-hsm.metacentrum.cz, symlink /storage/jihlava1-cerit/
storage-jihlava2-archive.metacentrum.cz /storage/jihlava2-archive 2 050 TiB 3
storage-liberec3-tul.metacentrum.cz /storage/liberec3-tul/ 30 TiB
storage-plzen1.metacentrum.cz /storage/plzen1/ 352 TB 2 (storage-eiger1|storage-eiger2|storage-eiger3).zcu.cz
storage-plzen2-archive.metacentrum.cz /storage/plzen2-archive/ zrušeno - decommissioned nfs.du1.cesnet.cz
storage-plzen3-kky.metacentrum.cz /storage/plzen3-kky/ 73 TiB 3
storage-praha1.metacentrum.cz /storage/praha1/ 100 TB 3 storage-praha1(a|b).metacentrum.cz
storage-praha4-fzu.metacentrum.cz /storage/praha4-fzu/ 15 TB
storage-praha5-elixir.metacentrum.cz /storage/praha5-elixir/ 157 TB 3

The often user problem is the inability to access the folders in /storage while they are logged in to frontend by the ssh key. In this case they don't have generated the Kerberos tickets. This can be repaired by invocation of kinit command.

Shared network file system AFS on all machines

All machines in MetaCentrum have AFS mounted. The system is secure and scalable to the whole world, but it is rather slow. You can mount the AFS file system on your own workstation if you want.

Access rights

All volumes are covered by standard set of unix permissions. Default set up of directories (including the home directory) and files is 755, ie. rwxr-xr-x. In case you want to change this behavior you can do it by invoking command

chmod 700 <directory>

To change the rights for your home directory and for computing/temporal directory (scratch) use

chmod 700 /home/$USER
chmod 700 $SCRATCHDIR

Changing the rights can be performed also from user scripts if you compute some sensitive data. On the other hand you can use current properties allowing read and execute to share your scripts in groups of users.

Quotas

Quotas are the mechanism for monitoring of disk storage capacity for administrators and for the error prevention. This error could negatively affect the work of all users because of possible fulfilment. You can check your current quotas at your "personal view of storages" page.