Running the jobs
Access to compute nodes is provided via a scheduler and an associated batch queuing application. To run an application, the user submits a 'job' to the scheduler. The scheduler will then determine, based on fairshare, available and requested resources and other jobs waiting in the queue, when the job should run. At that time, the scheduler hands the job to the queuing application which initiates the job.
The MetaCentrum batch systems uses a fairshare scheduling policy. Fairshare is a mechanism which allows historical resource utilization information to be incorporated into job feasibility and priority decisions.
- This scheduling tries to distribute the resources in a 'fair' way between groups and persons that are using the system. The submit-time of a job is corrected, it is not in the same order as the jobs were submitted.
- When the job belongs to a user that has used a lot of cpu time and memory in past few days, the submit-time of the job is set to a later point in time. On the other hand, if he used less resources, the submit-time is set at an earlier point in time.
- The memory of the system with respect to usage of the system in the past is limited and decreases with time: yesterday is more important than the day before yesterday and so on. The memory lasts typically 30 days. The more the user has used the last days, the lower the priority is.
- On the other hand, the user is prioritized for publications with acknowledgment to MetaCentrum/CERIT-SC. User with higher number of publications (set in our system) is prioritized before the user with smaller number of publication.
The life cycle of a job can be divided into four stages: creation, submission, execution, and finalization.
- Creation - Typically, a submit script is written to hold all of the parameters of a job. These parameters could include how long a job should run (walltime), what resources are necessary to run, and what to execute.
- Submission - A job is submitted with the qsub command. Submission script is a shell script that describes the processing to carry out (e.g. the application, its input and output, etc.) and requests computer resources (number of cpus, amount of memory) to use for processing. Directives are job specific requirements given to the job scheduler. The most important directives are those that request resources. The most common are the walltime limit (the maximum time the job is allowed to run) and the number of CPU, MEM, and SRATCH required to run the job.
- Execution - Job is running.
- Finalization - When a job completes any left over user processes are killed, and by default, the stdout and stderr files are copied to the directory where the job was submitted.
Backfill is a scheduling optimization which allows a scheduler to make better use of available resources by running jobs out of order (it allows to run short jobs in gaps).