I am running the WRF-4.2 weather model using openmpi-3.1.2 and pbs_version = 19.1.1, and I’d like to test my 10gBE interconnect. I had been relying on the backward compatibility of using (as an example of what worked) -l nodes=1:ppn=8. Now I can select nodes connected to the 10gBE switch using -l select=1:ncpus=8:switch=10gBE.
However, I find that I now must supply mpiprocs in my select statement. I’m not sure if that is required only when I use all the CPU cores on a node, or not. This worked:
-l select=1:ncpus=32:mpiprocs=32:switch=10gBE
These two caused PBS to start the job, but WRF failed:
-l select=2:ncpus=32:mpiprocs=64:switch=10gBE
-l select=4:ncpus=32:mpiprocs=128:switch=10gBE
They both had MPI_COMM_WORLD errors in MPI_Cart_create. I searched this forum, the WRF forum, and https://www.altair.com/pdfs/pbsworks/PBSAdminGuide19.2.3.pdf and found mentions of mpiprocs, but no definition.
What is it used for, and what should I be setting it to? The total number of CPU cores I want to use? The maximum per node? 1 per job (if it means the “mother” thread)?
Thanks!
We put resource definitions in the Resources chapter in the Reference Guide. Here’s what we say there:
Number of MPI processes for this chunk. Cannot use sum from chunks as job-wide limit. The number of lines in PBS_NODEFILE is the sum of the values of mpiprocs for all chunks requested by the job. For each chunk with mpiprocs=P, the host name for that chunk is written to the PBS_NODEFILE P times.
using 20 MPI ranks:
qsub -l select=2:ncpus=10:mpiprocs=10 -l place=scatter
and the job will use 2 nodes with 10 processes per node
Please run this command:
qsub -l select=2:ncpus=10:mpiprocs=10 -l place=scatter -I
you get a console on remote node, then run cat $PBS_NODEFILE
qsub -l select=1:ncpus=20:mpiprocs=20
then the job will use 1 node with all 20 processes running on that node. Suppose this job takes 1800 seconds to complete.
qsub -l select=1:ncpus=20:mpiprocs=20 -I
you get a console on remote node, then run cat $PBS_NODEFILE
Please vary the mpiprocs attribute and see how it references proccess list to be spawned across the host in $PBS_NODEFILE
mpiprocs - number of mpi processes or core per node