Running a job with multi-threaded processes

Users/Site Administrators
1

Hello all, new PBS user here.

I have an application that makes use of multi-threading. If I run the application, the process will create multiple threads and consume multiple cores. The application also utilizes OpenMPI so that we can run multiple instances of the application on separate machines in order to increase the application’s throughput. We would like to be able to use PBS to schedule these jobs. Typically, each process will consume 12 cores and we run one process per node. However, if a machine has a large number of cores, we could run multiple processes on one node.

As is, I can submit a job to PBS and PBS will limit the number of running processes to the number of cores available on each my nodes. This works fine if a process is single-threaded. However, in my case, because my application is multi-threaded, this creates excessive CPU contention.

Is there any way to tell PBS (via qsub or some other mechanism) that each executing process consumes a certain number of cores and have PBS limit the number of processes that can run on a node accordingly? So if my process uses 12 threads and a node has 24 cores, a maximum of 2 processes would run on that node.

If this is not possible, is there a way to limit each node to only a single process? So if I have 8 nodes, I could submit 2 jobs that each use 4 nodes and have them distributed between the nodes. If I were to submit another 4-node job, that job would be queued until 4 nodes became available.

Thanks

2

Hello john_m,

For your first case (1 process using 12 threads, 1 core per thread) you could submit a job like:

qsub -l select=1:ncpus=12

This will give you one “chunk” (see section 4.2.2 Glossary in the 2020.1 PBS User Guide) of 12 cores. PBS will assign a chunk to a single node. Further, if possible (e.g. there are 24 cores per node) PBS will assign multiple chunks to the same node. Also, on the more recent versions PBS will use cgroups to restrict the process to only those resources requested in the qsub.

If running OpenMP jobs if is useful to note that by default PBS sets the value of the environment variable OMP_NUM_THREADS to the number of ncpus requested (see section 5.1.7 MPI Environment Variables in the 2020.1 PBS User Guide). If you want to run more or less than 1 thread per core you could specify ompthreads on your chunk. For example (explicitly stating the default):

qsub -l select=1:ncpus=12:ompthreads=12

You can also specify the number of mpi processes to run on a chunk using the attribute mpiprocs (see section 5.1.3 Specifying Number of MPI Processes Per Chunk in the 2020.1 PBS User Guide). To specify the same chunk running 12 mpi processes:

qsub -l select=1:ncpus=12:mpiprocs=12

This will affect the number of times the host is listed in the nodes file (see section 5.1.2 The Job Node File in the 2020.1 PBS User Guide). Since you’re running 1 mpi process on a chunk you would say (which also happens to be the default):

qsub -l select=1:ncpus=12:mpiprocs=1

If you want to run a hybrid job using multiple threads on one mpi process you could request (which also happens to be the default):

qsub -l select=1:ncpus=12:ompthreads=12:mpiprocs=1

If you want to specify the resources for multiple mpi processes you would request multiple chunks. For example:

qsub -l select=2:ncpus=12:ompthreads=12:mpiprocs=1

would give your job 2 chunks (one for each MPI process) using 12 ncpus (cores), one OpenMPI thread per core, 1 MPI process per chunk.

Hope this helps!

3

Thanks very much for reply and the detailed explanation, smgoosen. This indeed does help.

I’ve seen many examples of select statements, but none seemed to apply to my situation. Most examples give a 1:1 correspondence between ncpus and mpiprocs. It’s now clear why this does not apply to all situations.