# Mpiprocs definition?

**URL:** <https://community.openpbs.org/t/mpiprocs-definition/2195>\
**Category:** Users/Site Administrators\
**Created:** [July 3, 2020, 2:31pm UTC](https://community.openpbs.org/t/mpiprocs-definition/2195 "2020-07-03T14:31:47Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![bartbrashers](https://avatars.discourse-cdn.com/v4/letter/b/da6949/32.png) [@bartbrashers](https://community.openpbs.org/u/bartbrashers)\
**Post date:** [July 3, 2020, 2:31pm UTC](https://community.openpbs.org/t/mpiprocs-definition/2195/1 "2020-07-03T14:31:47Z")

</div>

I am running the WRF-4.2 weather model using openmpi-3.1.2 and pbs\_version = 19.1.1, and I’d like to test my 10gBE interconnect. I had been relying on the backward compatibility of using (as an example of what worked) **-l nodes=1:ppn=8**. Now I can select nodes connected to the 10gBE switch using **-l select=1:ncpus=8:switch=10gBE**.

However, I find that I now must supply **mpiprocs** in my select statement. I’m not sure if that is required only when I use all the CPU cores on a node, or not. This worked:

```
-l select=1:ncpus=32:mpiprocs=32:switch=10gBE

```

These two caused PBS to start the job, but WRF failed:

```
-l select=2:ncpus=32:mpiprocs=64:switch=10gBE
-l select=4:ncpus=32:mpiprocs=128:switch=10gBE

```

They both had MPI\_COMM\_WORLD errors in MPI\_Cart\_create. I searched this forum, the WRF forum, and [https://www.altair.com/pdfs/pbsworks/PBSAdminGuide19.2.3.pdf](https://www.altair.com/pdfs/pbsworks/PBSAdminGuide19.2.3.pdf) and found mentions of mpiprocs, but no _definition_.

What is it used for, and what should I be setting it to? The total number of CPU cores I want to use? The maximum per node? 1 per job (if it means the “mother” thread)?

Thanks!

---

<div class="post-metadata">

**Author:** ![agurban](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/agurban/32/67_2.png) [@agurban](https://community.openpbs.org/u/agurban)\
**Post date:** [July 3, 2020, 11:11pm UTC](https://community.openpbs.org/t/mpiprocs-definition/2195/2 "2020-07-03T23:11:30Z")

</div>

We put resource definitions in the Resources chapter in the Reference Guide. Here’s what we say there:  
Number of MPI processes for this chunk. Cannot use sum from chunks as job-wide limit. The number of lines in PBS\_NODEFILE is the sum of the values of mpiprocs for all chunks requested by the job. For each chunk with mpiprocs=P, the host name for that chunk is written to the PBS\_NODEFILE P times.

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [July 6, 2020, 8:42am UTC](https://community.openpbs.org/t/mpiprocs-definition/2195/3 "2020-07-06T08:42:38Z")

</div>

using 20 MPI ranks:

qsub -l select=2:ncpus=10:mpiprocs=10 -l place=scatter  
and the job will use 2 nodes with 10 processes per node

Please run this command:  
qsub -l select=2:ncpus=10:mpiprocs=10 -l place=scatter -I

you get a console on remote node, then run cat $PBS\_NODEFILE

qsub -l select=1:ncpus=20:mpiprocs=20  
then the job will use 1 node with all 20 processes running on that node. Suppose this job takes 1800 seconds to complete.

qsub -l select=1:ncpus=20:mpiprocs=20 -I  
you get a console on remote node, then run cat $PBS\_NODEFILE

Please vary the mpiprocs attribute and see how it references proccess list to be spawned across the host in $PBS\_NODEFILE

mpiprocs - number of mpi processes or core per node