Working with data/Working with data in a job

Z MetaCentrum
Přejít na: navigace, hledání

(Czech version)

Related topics
How to compute
Types of data storages
Types of scratch storage

This page provide information about preparing and using data within a job. If you would like to learn more about MetaCentrum file systems, read the Types of data storages page.

Data preparation

1. Log on any available frontend.

2. Select one the options below:

Data and jobs

Before you begin, please keep in mind these rules:

  • Scratch storage is the best storage type for data in a running job
  • /storage volumes are best storage types for keeping and saving your data between running jobs
  • Archive data are handled by CESNET Data Department, see Backup and archiving for more information
  • Current directory is saved into PBS_O_WORKDIR variable when you submit your job via qsub command. You can find job's outputs there.
  • It is always better to use the absolute path rather than working directory

One processor job example

This example shows how to set up your job to work with scratch directory.

1. Prepare your input data, e.g. to /storage/brno2/home/$LOGNAME/myJob directory

2. Prepare your batch script

ZarovkaMala.png Note: This example deletes data of unsuccessful jobs. Add the following code behind trap 'clean_scratch' TERM EXIT line, if you would like to keep the data :

trap 'cp -r $SCRATCHDIR/run.log $SCRATCHDIR/uncomplete.data $DATADIR && clean_scratch' TERM

Create myJob.sh text file in your favourite text editor with following lines:

#!/bin/bash
#PBS -N oneCPUjob
#PBS -l walltime=4:00:00
#PBS -l select=1:ncpus=2:mem=6gb:scratch_local=12gb
#PBS -j oe
#PBS -m e
# Options for scheduling system - job will last 4 hours at maximum, 1 machine with 2 processor + 6gb memory RAM + 12gb scratch storage is requested 
# Error output and standard output will be merged (-j oe), a notification email will be sent when the job is finished (-m e)
# You can adjust these options by your needs
 
trap 'clean_scratch' TERM EXIT # sets up scratch cleaning in case an error occurs

module add target_application # loads desired application

DATADIR="/storage/brno2/home/$LOGNAME/myJob/" # sets up your input data directory


if [ ! -d "$SCRATCHDIR" ] ; then echo "Scratch directory is not created!" 1>&2; exit 1; fi #checks if scratch directory is created

cp $DATADIR/data $SCRATCHDIR/ # copies data from the input data directory to your scratch directory
cd $SCRATCHDIR # moves to your scratch diretory

#Space for your computation:
#-----

#-----


echo "uloha $PBS_JOBID na `uname -n`" >result # just an example computation

cp result $DATADIR || export CLEAN_SCRATCH=false # copies result to your input data directory. The result will not be removed if the copying fails.

3. Submit the job

qsub myJob.sh

You can find your result in /storage/brno2/home/$LOGNAME/myJob directory when the job is finished.

How to check the status of your jobs

  • visit assigned node for your job via ssh (e.g. ssh user_name@luna43.fzu.cz)
  • check /var/spool/pbs/spool/JOBID.{OU,ER} (e.g. more XY.arien/wagap-pro.ics.muni.cz.OU/ER)