When is it appropriate to add mpiprocs= when using #PBS -l select= to request resources? Like #PBS -l select=4:ncpus=32:mpiprocs=32 or #PBS -l select=4:ncpus=32 .
It basically depends on application and MPI in conjunction with what applicaiton vendor suggests about speedup , scalability , performance improvements, based on that you adjust the job requests.
PureMPI – #PBS -l select=4:ncpus=32:mpiprocs=32
Hybrid MPI ( MPI / OpenMP) – #PBS -l select=4:ncpus=32:mpiprocs=4:ompthreads=8 (caution: mpiprocs × ompthreads = ncpus)
Memory Intensive ( MPI) – #PBS -l select=2:ncpus=32:mpiprocs=16:mem=128gb
- select=4:ncpus=32 -l place=scatter: 4 nodes, 32 cores each. MPI behavior is undefined (depends on site defaults). Simple scripts or non-MPI software.
- select=4:ncpus=32:mpiprocs=32 -l place=scatter: 4 nodes, 32 MPI ranks per node. Pure MPI (e.g., standard CFD or FEA solvers).
- select=4:ncpus=32:mpiprocs=1 -l place=scatter: 4 nodes, 1 MPI rank per node.
Usually, the application user benchmarks their model or workflow based on the suggestion given by the application vendor and fine tunes these combinations.
Always use mpiprocs for MPI jobs.
What is the role of the mpiprocs parameter? Whether to use MPI should depend on the program actually running on the computing nodes in the job script. For example, if I use #PBS -l select=4:ncpus=32, this will allocate 32 CPUs for me to use. If I run an MPI program, it will still start MPI processes, regardless of whether I use the mpiprocs parameter or not
The job of mpiprocs is to tell PBS exactly how many MPI processes or ranks you want per chunk. PBS needs this number to generate $PBS_NODEFILE and set up your parallel layout. Since you did not include mpiprocs, PBS doesn’t know your intended rank count. This would not be an issue, though, assuming your MPI launcher is properly configured to detect and use the allocated hosts and slot.
Please check the contents of your $PBS_NODEFILE with and without using mpiprocs in your select statement, to see the difference.