Working with data/Direct access to data storages

Z MetaCentrum
Přejít na: navigace, hledání

You can access data storages directly which means you don't have to use frontends or any other machines. This method is also highly recommended for working with huge amount of data. These are available access options of direct accessing:

List of NSF4 servers: Zálohovací třídy jsou popsány v / Back-up policy is described at: Politika_zálohování (Back-up policy). Výtah/summary:

  • třída 2 - záloha (pouze) formou časových řezů / class 2 - backup (only) in a form of time slices
  • třída 3 - data se záložní kopií / class 3 - data with a backup copy
NFS4 server adresář - directory velikost - capacity zálohovací třída - back-up policy alternativní jména serverů v Perunovi - alternative name / poznámka - note
storage-brno1-cerit.metacentrum.cz /storage/brno1-cerit/ 1.8 PB 2 nfs-ntc.ics.muni.cz
storage-brno2.metacentrum.cz /storage/brno2/ 110 TB 3 (nienna1|nienna2|nienna-home).ics.muni.cz
storage-brno3-cerit.metacentrum.cz /storage/brno3-cerit/ 932 TB 2 nfs-kat.cerit-sc.cz
storage-brno4-cerit-hsm.metacentrum.cz /storage/brno4-cerit-hsm/ zrušeno - decommissioned data archived in /storage/brno1-cerit/
storage-brno5-archive.metacentrum.cz /storage/brno5-archive/ 5 387 TiB 3 nfs.du3.cesnet.cz
storage-brno6.metacentrum.cz /storage/brno6/ 262 TB 2
storage-brno7-cerit.metacentrum.cz /storage/brno7-cerit/ ruší se - decommissioned 2 data archived in /storage/brno1-cerit/
storage-brno8.metacentrum.cz /storage/brno8/ 88 TB 3 in past /storage/ostrava1/
storage-brno9-ceitec.metacentrum.cz /storage/brno9-ceitec/ 262 TB 3 storage-ceitec1.ncbr.muni.cz - pro NCBR CEITEC
storage-brno10-ceitec-hsm.metacentrum.cz /storage/brno10-ceitec-hsm/ 3 dedicated to NCBR CEITEC
storage-brno11-elixir.metacentrum.cz /storage/brno11-elixir/ 313 TB 2 dedicated to ELIXIR-CZ
storage-budejovice1.metacentrum.cz /storage/budejovice1/ 44 TB 3 (storage-cb1|storage-cb2).metacentrum.cz
storage-jihlava1-cerit.metacentrum.cz /storage/jihlava1-cerit/ zrušeno - decommissioned data archived to /storage/brno4-cerit-hsm/fineus, storage-brno4-cerit-hsm.metacentrum.cz, symlink /storage/jihlava1-cerit/
storage-jihlava2-archive.metacentrum.cz /storage/jihlava2-archive 2 050 TiB 3
storage-liberec3-tul.metacentrum.cz /storage/liberec3-tul/ 30 TiB
storage-plzen1.metacentrum.cz /storage/plzen1/ 352 TB 2 (storage-eiger1|storage-eiger2|storage-eiger3).zcu.cz
storage-plzen2-archive.metacentrum.cz /storage/plzen2-archive/ zrušeno - decommissioned nfs.du1.cesnet.cz
storage-plzen3-kky.metacentrum.cz /storage/plzen3-kky/ 73 TiB 3
storage-praha1.metacentrum.cz /storage/praha1/ 100 TB 3 storage-praha1(a|b).metacentrum.cz
storage-praha4-fzu.metacentrum.cz /storage/praha4-fzu/ 15 TB
storage-praha5-elixir.metacentrum.cz /storage/praha5-elixir/ 157 TB 3

SFTP

Windows users

Windows users need an SFTP client, we recommend the WinSCP application. You can follow the Usage of WinSCP tutorial but you have to fill in target NFS4 server instead of frontend in the Step 1, also make sure you have selected SFTP file protocol.

Linux users

You can use your terminal and sftp command as shown below:

ZarovkaMala.png Note: You can find additional information about SFTP command here

sftp 'META username'@target_NFS4_server # Login
help # Shows available commands
get target_file # Downloads target file to your local system
get -r target_directory # Downloads target directory to your local system
put target_file # Uploads target file to server
put -r target_directory # Uploads target directory to server

The example below includes a bug affecting Ubuntu 14.04+. Create the target directory on the server first to work around this issue. SFTP1.png

SCP

SCP option is described in the Data transfer topic.

SSH

Access via SSH is possible for advanced users with special needs. Selected programs, serving for data manipulation directly at the NFSv4 server operating disk storage, can be run through SSH. Data can be moved from the user machine to the disk storage (in addition to SCP input/output of a locally run program can be redirected to/from a file at storage).

Commands manipulating with large amounts of data run at the NFSv4 server much faster compared to the same operation at a client machine (e.g. cluster node). On the other hand these operations can easily overload NFSv4 server. If you plan massive file moves, contact us in advance, please.

NOTE: The full shell is not available via SSH, and it is impossible to run programs stored at users volume. No computation should be run at the NFSv4 server. Only the following commands are available:

  • /usr/bin/scp,
  • /usr/lib/sftp-server,
  • /bin/cp,
  • /bin/ls,
  • /bin/tar,
  • /bin/cat,
  • /bin/dd,
  • /bin/rm,
  • /bin/mkdir,
  • /bin/mv,
  • /bin/rmdir,
  • /bin/chmod,
  • /usr/bin/gzip,
  • /usr/bin/gunzip

When copying files with dd set block size (bs parameter) to at least 1M. Operation will be faster (comparing with standart value 512 byte).

Example

Listing content of user home directory "login" from remote machine by the following command:

ssh login@depot1.mc.cesnet.cz ls -l

Full path can be used as well:

ssh login@depot1.mc.cesnet.cz ls -l /storage/home/login

ZarovkaMala.png Note: No shell is available, so command ssh user@depot1.mc.cesnet.cz will be refused by the server