Mpiprocs definition?

I am running the WRF-4.2 weather model using openmpi-3.1.2 and pbs_version = 19.1.1, and I’d like to test my 10gBE interconnect. I had been relying on the backward compatibility of using (as an example of what worked) -l nodes=1:ppn=8. Now I can select nodes connected to the 10gBE switch using -l select=1:ncpus=8:switch=10gBE.

However, I find that I now must supply mpiprocs in my select statement. I’m not sure if that is required only when I use all the CPU cores on a node, or not. This worked:

-l select=1:ncpus=32:mpiprocs=32:switch=10gBE

These two caused PBS to start the job, but WRF failed:

-l select=2:ncpus=32:mpiprocs=64:switch=10gBE
-l select=4:ncpus=32:mpiprocs=128:switch=10gBE

They both had MPI_COMM_WORLD errors in MPI_Cart_create. I searched this forum, the WRF forum, and and found mentions of mpiprocs, but no definition.

What is it used for, and what should I be setting it to? The total number of CPU cores I want to use? The maximum per node? 1 per job (if it means the “mother” thread)?


We put resource definitions in the Resources chapter in the Reference Guide. Here’s what we say there:
Number of MPI processes for this chunk. Cannot use sum from chunks as job-wide limit. The number of lines in PBS_NODEFILE is the sum of the values of mpiprocs for all chunks requested by the job. For each chunk with mpiprocs=P, the host name for that chunk is written to the PBS_NODEFILE P times.

using 20 MPI ranks:

qsub -l select=2:ncpus=10:mpiprocs=10 -l place=scatter
and the job will use 2 nodes with 10 processes per node

Please run this command:
qsub -l select=2:ncpus=10:mpiprocs=10 -l place=scatter -I

you get a console on remote node, then run cat $PBS_NODEFILE

qsub -l select=1:ncpus=20:mpiprocs=20
then the job will use 1 node with all 20 processes running on that node. Suppose this job takes 1800 seconds to complete.

qsub -l select=1:ncpus=20:mpiprocs=20 -I
you get a console on remote node, then run cat $PBS_NODEFILE

Please vary the mpiprocs attribute and see how it references proccess list to be spawned across the host in $PBS_NODEFILE

mpiprocs - number of mpi processes or core per node