Batch jobs

From HP-SEE Wiki

Contents 1 Maui/Torque 2 Condor-HTC 3 Sun Grid Engine 4 IBM Loadleveler

Maui/Torque

Condor-HTC

Sun Grid Engine

Section contributed by NIIFI

SGE is typically used on computer farms, or high-performance computing clusters, and is responsible for accepting, scheduling, dispatching, and managing the remote and distributed execution of large numbers of standalone, parallel or interactive user jobs. It also manages and schedules the allocation of distributed resources such as processors, memory, disk space, and software licenses. Several commands help the users to manage the scheduler system. Here we collected the most useful ones.

Resource status commands:

• cluster queue summary: qstat –g c
• resource availability per host: qhost –F
• show the parallel environments: qconf –spl
• show the available queues: qconf -sql

Job managment commands:

• query the job status of all jobs: qstat –u \*
• submit a job: qsub script.sh
• job status query: qstat –j JOB_ID
• query the reason why out job has been not scheduled yet: qalter -w v JOB_ID

Example MPI submit script:

 #!/bin/bash
 #$ -N CONNECTIVITY
 #$ -pe mpi 10
 mpirun -np $NSLOTS ./connectivity -v

The NSLOTS variable will contain that slots number what we requested for the parallel environment (10). The following table show the most common qsub parameters.

qsub parameter	Example	Meaning
-N name	-N test	The job name
-cwd	-cwd	The output and the error files will be created in the actual directory
-S shell	-S /bin/bash	The shell type
-j {y|n}	-j y	Concate the error and output files
-r {y|n}	-r y	Job should be or not restarted after restart
-M mail	-M something@example.org	Job state information will be send to this mail address
-m {b|e|a|s|n}	-m bea	The requested job states will be reported to the mail address
-l	-l h_cpu=0:15:0	Wall time limit for the job
-l	-l h_vmem=1G	Memory limit for the job
-pe	-pe mpi 10	This is site specific parameter which setup the requested parallel environment

There is a possibility to send array jobs to SGE. For example, you may have 1000 data sets, and you want to run a single program on them, using the cluster. Here is an example to send an array job:

 qsub -t 1-1000 array.sh

array.sh:

 #!/bin/sh
 #$ -N PI_ARRAY_TEST
 ./pi `expr $SGE_TASK_ID \* 100000`

The SGE_TASK_ID is an inner variable which is set by one by one in the jobs.

REFERENCES

IBM Loadleveler

Section contributed by IICT-BAS

Jobs in BlueGene/P are scheduled for execution by system called LoadLeveler. The prepared jobs are submitted for execution via the command “llsubmit”, which receives something called Job Control File, which describes the executing program and its environment. This puts the job into a queue of waiting jobs. There are scheduling strategies via which the jobs are prioritized. When suitable resource is available, the next task in the queue will execute. You can see the contents of the queue with “llq”. If the status of the job shows that there is a problem, it is need to look at the error output and remove your job via “llcancel”. If it is necessary to copy the whole user environment when the job is running, command “environment=COPY_ALL” will do it. LoadLeveler generate JCF (job control file) file.

Example for JCF, named hello.jcf :

# @ job_name = hello
# @ comment = "This is a HelloWorld program"
# @ error = $(jobid).err
# @ output = $(jobid).out
# @ environment = COPY_ALL;
# @ wall_clock_limit = 01:00:00
# @ notification = never
# @ job_type = bluegene
# @ bg_size = 128
# @ class = n0128
# @ queue
/bgsys/drivers/ppcfloor/bin/mpirun -exe hello -verbose 1 -mode VN -np 512

To send it for execution:

llsubmit hello.jcf

There are several important parameters:

Parameter	Meaning
# @ job_name	The name for the job, could be anything.
# @ comment	Some comment.
# @ error	Where to send the stderr. Writing to file descriptor 1 (in C) writes to this file.
# @ output	Same here, but for the stdout (file descriptor 0 in C).
# @ environment	This instructs LoadLeveler to copy the whole user environment when running the job. Thus, the job has the same environment as the user that executes llsubmit.
# wall_clock_limit	Time limit; after this time, the job is cancelled automatically. This cannot be more than a certain time limit imposed by the class of the job.
# @ notification	There is no infrastructure for notifications, so 'never' is a good value for this parameter.
# @ job_type	This MUST be bluegene.
# @ bg_size	This must be an integer, divisable by 128, but not larger than 2048. This gives the number of computing nodes that will be used in order to execute the job.This must correspond to the class of the job.
# @ class	This is the class of the job. The most important parameter. Different classes have different priorities.
# @ queue	This instructs LoadLeveler to put the job in the queue.