Batch jobs
From HP-SEE Wiki
Contents |
Maui/Torque
Condor-HTC
Sun Grid Engine
Section contributed by NIIFI
SGE is typically used on computer farms, or high-performance computing clusters, and is responsible for accepting, scheduling, dispatching, and managing the remote and distributed execution of large numbers of standalone, parallel or interactive user jobs. It also manages and schedules the allocation of distributed resources such as processors, memory, disk space, and software licenses. Several commands help the users to manage the scheduler system. Here we collected the most useful ones.
Resource status commands:
- cluster queue summary: qstat –g c
- resource availability per host: qhost –F
- show the parallel environments: qconf –spl
- show the available queues: qconf -sql
Job managment commands:
- query the job status of all jobs: qstat –u \*
- submit a job: qsub script.sh
- job status query: qstat –j JOB_ID
- query the reason why out job has been not scheduled yet: qalter -w v JOB_ID
Example MPI submit script:
#!/bin/bash #$ -N CONNECTIVITY #$ -pe mpi 10 mpirun -np $NSLOTS ./connectivity -v
The NSLOTS variable will contain that slots number what we requested for the parallel environment (10). The following table show the most common qsub parameters.
qsub parameter | Example | Meaning |
---|---|---|
-N name | -N test | The job name |
-cwd | -cwd | The output and the error files will be created in the actual directory |
-S shell | -S /bin/bash | The shell type |
-j {y|n} | -j y | Concate the error and output files |
-r {y|n} | -r y | Job should be or not restarted after restart |
-M mail | -M something@example.org | Job state information will be send to this mail address |
-m {b|e|a|s|n} | -m bea | The requested job states will be reported to the mail address |
-l | -l h_cpu=0:15:0 | Wall time limit for the job |
-l | -l h_vmem=1G | Memory limit for the job |
-pe | -pe mpi 10 | This is site specific parameter which setup the requested parallel environment |
There is a possibility to send array jobs to SGE. For example, you may have 1000 data sets, and you want to run a single program on them, using the cluster. Here is an example to send an array job:
qsub -t 1-1000 array.sh
array.sh:
#!/bin/sh #$ -N PI_ARRAY_TEST ./pi `expr $SGE_TASK_ID \* 100000`
The SGE_TASK_ID is an inner variable which is set by one by one in the jobs.
REFERENCES
IBM Loadleveler
Section contributed by IICT-BAS
Jobs in BlueGene/P are scheduled for execution by system called LoadLeveler. The prepared jobs are submitted for execution via the command “llsubmit”, which receives something called Job Control File, which describes the executing program and its environment. This puts the job into a queue of waiting jobs. There are scheduling strategies via which the jobs are prioritized. When suitable resource is available, the next task in the queue will execute. You can see the contents of the queue with “llq”. If the status of the job shows that there is a problem, it is need to look at the error output and remove your job via “llcancel”. LoadLeveler generate JCF (job control file) file.
Example for JCF, named hello.jcf :
# @ job_name = hello # @ comment = "This is a HelloWorld program" # @ error = $(jobid).err # @ output = $(jobid).out # @ environment = COPY_ALL; # @ wall_clock_limit = 01:00:00 # @ notification = never # @ job_type = bluegene # @ bg_size = 128 # @ class = n0128 # @ queue /bgsys/drivers/ppcfloor/bin/mpirun -exe hello -verbose 1 -mode VN -np 512
To send it for execution:
llsubmit hello.jcf
There are some parameters, that can be added to the sending command:
-exe <executable_file> - the executable file itself; -args "<arguments>" - arguments to the executable file; -verbose 1 - write information about job startup/finalizing in the stderr file -mode VN|SMP|DUAL - provides the mode of execution; -np N - the number of processes on which the job will execute; -env BG_MAXALIGNEXP=-1 – very important parameter, which instructs the CN kernel to ignore alignment traps.
There are several important parameters, that can be used in JCF file:
Parameter | Meaning |
---|---|
# @ job_name | The name for the job, could be anything. |
# @ comment | Some comment. |
# @ error | Where to send the stderr. Writing to file descriptor 1 (in C) writes to this file. |
# @ output | Same here, but for the stdout (file descriptor 0 in C). |
# @ environment | This instructs LoadLeveler to copy the whole user environment when running the job. Thus, the job has the same environment as the user that executes llsubmit. |
# @ wall_clock_limit | Time limit; after this time, the job is cancelled automatically. This cannot be more than a certain time limit imposed by the class of the job. |
# @ notification | There is no infrastructure for notifications, so 'never' is a good value for this parameter. |
# @ job_type | This MUST be bluegene. |
# @ bg_size | This must be an integer, divisable by 128, but not larger than 2048. This gives the number of computing nodes that will be used in order to execute the job.This must correspond to the class of the job. |
# @ class | This is the class of the job. The most important parameter. Different classes have different priorities. |
# @ queue | This instructs LoadLeveler to put the job in the queue. |