# Openmpi support

**URL:** <https://community.openpbs.org/t/openmpi-support/1396>\
**Category:** Users/Site Administrators\
**Created:** [January 14, 2019, 3:17am UTC](https://community.openpbs.org/t/openmpi-support/1396 "2019-01-14T03:17:17Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![liuxiaopiao](https://avatars.discourse-cdn.com/v4/letter/l/65b543/32.png) [@liuxiaopiao](https://community.openpbs.org/u/liuxiaopiao)\
**Post date:** [January 14, 2019, 3:17am UTC](https://community.openpbs.org/t/openmpi-support/1396/1 "2019-01-14T03:17:17Z")

</div>

I have one question related to openmpi support.  
My cluster is setup by the below steps:

1. install openmpi by “sudo yum install openmpi openmpi-devel”.
2. module load mpi/openmpi-x86\_64
3. install pbs server/execution rpm package in node1,node2.

And according to [FAQ: Running jobs under Torque / PBS Pro](https://www.open-mpi.org/faq/?category=tm#tm-configure-ompi), I checked the openmpi and pbs by

> [test@pbspro-server ~]$ ompi\_info | grep ras  
> MCA ras: gridengine (MCA v2.0.0, API v2.0.0, Component v1.10.7)  
> MCA ras: loadleveler (MCA v2.0.0, API v2.0.0, Component v1.10.7)  
> MCA ras: simulator (MCA v2.0.0, API v2.0.0, Component v1.10.7)  
> MCA ras: slurm (MCA v2.0.0, API v2.0.0, Component v1.10.7)

Does this mean “my openmpi didn’t support PBS”? But it seemed that I can use “mpirun” in pbs job scripts.

Do I need to install openmpi by compiling the code with " --with-tm option"?

Thanks a lot!

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [January 14, 2019, 10:01am UTC](https://community.openpbs.org/t/openmpi-support/1396/2 "2019-01-14T10:01:12Z")

</div>

> [@liuxiaopiao](#):
>
> Does this mean “my openmpi didn’t support PBS”? But it seemed that I can use “mpirun” in pbs job scripts.

PBS Supports openmpi, but the openmpi should be compiled with PBS TM option.

> [@liuxiaopiao](#):
>
> Do I need to install openmpi by compiling the code with " --with-tm option"?

Thats correct, for this to work you need to compile openmpi with PBS TM , please follow the below

> [@Compile OpenMPI with PBSpro 14.1.10](http://community.openpbs.org/t/compile-openmpi-with-pbspro-14-1-10/159/11):
>
> Adding a pbs\_config script is one way of making openmpi’s configure to succeed, but here also a simple manual way to do it: export LD\_LIBRARY\_PATH=/opt/pbs/lib:$LD\_LIBRARY\_PATH export LDFLAGS="-L/opt/pbs/lib -lpbs -lpthread -lcrypto" ./configure --with-tm=/opt/pbs This should simply configure openmpi with PBS tm support. I tried with 2.0.2 as well.

---

<div class="post-metadata">

**Author:** ![liuxiaopiao](https://avatars.discourse-cdn.com/v4/letter/l/65b543/32.png) [@liuxiaopiao](https://community.openpbs.org/u/liuxiaopiao)\
**Post date:** [January 14, 2019, 2:37pm UTC](https://community.openpbs.org/t/openmpi-support/1396/3 "2019-01-14T14:37:28Z")

</div>

Thanks very much for the info. Is there any simple way to check if the openmpi works well with my pbs cluster?  
Or is there any simple script to help verify openmpi with pbs?

Before i didn’t compile openmpi with tm option, I still could run mpirun hostname inside pbs script, so I would like to have one golden way to check after I build openmpi with tm option to see if it is working fine with pbs cluster.

---

<div class="post-metadata">

**Author:** ![mkaro](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/mkaro/32/85_2.png) [@mkaro](https://community.openpbs.org/u/mkaro)\
**Post date:** [January 14, 2019, 4:42pm UTC](https://community.openpbs.org/t/openmpi-support/1396/4 "2019-01-14T16:42:34Z")

</div>

Here’s a draft tutorial I put together a couple weeks ago, intended for developers. It is a draft, so please be cautious of the commands you are running, and please ensure any critical data is backed up. It was not intended for production systems, but it should work anywhere. Please let us know if you find any problems with it. A couple things to note:

- You do not need to create the PTL RPM (for testing) if you don’t want to
- You may want to use the 18.1.3 release rather than the 19.1.1 beta. An issue was found with the beta where the database did not start.

[https://pbspro.atlassian.net/wiki/spaces/PBSPro/pages/998604803/Building+and+Installing+PBS+19+and+OpenMPI+Tutorial](https://pbspro.atlassian.net/wiki/spaces/PBSPro/pages/998604803/Building+and+Installing+PBS+19+and+OpenMPI+Tutorial)

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [January 14, 2019, 8:38pm UTC](https://community.openpbs.org/t/openmpi-support/1396/5 "2019-01-14T20:38:21Z")

</div>

- @mkaro suggestion - recommended

> [@liuxiaopiao](#):
>
> Before i didn’t compile openmpi with tm option, I still could run mpirun hostname inside pbs script, so I would like to have one golden way to check after I build openmpi with tm option to see if it is working fine with pbs cluster.

You can run openmpi with PBS Pro without compiling with PBS TM option, but it will be loose integration and the communication and process launch will be via ssh and upon job deletion (job clean up is not correct) , zombie process might be created which needs to be cleaned up manually (Otherwise, pbs\_attach would also help ) and usage data respect to each of the processes is not accounted.

If you use the MPI compiled with PBS TM , then the spawned processes are tightly controlled by PBS mom, proper accounting of the usage and job cleanup is guarantied.

**How to compile?**

> #wget [https://download.open-mpi.org/release/open-mpi/v4.0/openmpi-4.0.0.tar.bz2](https://download.open-mpi.org/release/open-mpi/v4.0/openmpi-4.0.0.tar.bz2)  
> #bzip2 -d openmpi-4.0.0.tar.bz2  
> #tar -xvf openmpi-4.0.0.tar  
> #cd openmpi-4.0.0  
> #export LD\_LIBRARY\_PATH=/opt/pbs/lib:$LD\_LIBRARY\_PATH  
> #export LDFLAGS=“-L/opt/pbs/lib -lpbs -lpthread -lcrypto”  
> #./configure --prefix=/path/to/shared/openmpi400 --with-tm=/opt/pbs/default --enable-mpi-interface-warning --enable-shared --enable-static --enable-cxx-exceptions  
> #make; make install

**How to run ?**  
#!/bin/bash  
#PBS -N pbs-openmpi-sh  
#PBS -l select=2:ncpus=4:mpiprocs=4  
#PBS -l place=scatter  
/opt/openmpi400/bin/mpirun -np `cat $PBS_NODEFILE | wc -l` /bin/hostname

```
qsub pbs-openmpi.sh

```

---

<div class="post-metadata">

**Author:** ![liuxiaopiao](https://avatars.discourse-cdn.com/v4/letter/l/65b543/32.png) [@liuxiaopiao](https://community.openpbs.org/u/liuxiaopiao)\
**Post date:** [January 15, 2019, 1:31pm UTC](https://community.openpbs.org/t/openmpi-support/1396/6 "2019-01-15T13:31:21Z")

</div>

Thanks @adarsh and @mkaro a lot for the detailed info.  
I have both tried your two ways to build openmpi with TM support successfully.

> `[test` `@pbspro` `-server bin]$ ./ompi_info | grep ras`  
> ` ` `MCA ras: loadleveler (MCA v2.` `0.0` `, API v2.` `0.0` `, Component v1.` `10.7` `)`  
> ` ` `MCA ras: simulator (MCA v2.` `0.0` `, API v2.` `0.0` `, Component v1.` `10.7` `)`  
> ` ` `MCA ras: slurm (MCA v2.` `0.0` `, API v2.` `0.0` `, Component v1.` `10.7` `)`  
> ` ` `MCA ras: tm (MCA v2.` `0.0` `, API v2.` `0.0` `, Component v1.` `10.7` `)`

The problem I met is that **the pbs jobs which are used to test PBS and openmpi integration have hanged**.

The steps are as follows:

1. the job scripts:

> [test@pbspro-server ~]$ cat test.sh  
> #!/bin/bash  
> #PBS -N pbs-openmpi-sh  
> #PBS -l select=2  
> #PBS -l place=scatter  
> cat $PBS\_NODEFILE  
> hostnumber=$(cat $PBS\_NODEFILE | wc -l)  
> /opt/openmpi/1.10.7/bin/mpirun hostname

[test@pbspro-server ~]$ cat test3.sh  
#PBS -l select=2  
#PBS -j oe  
/opt/openmpi/1.10.7/bin/mpirun ~/hello\_mpi

1. the job submission commands:  
[test@pbspro-server ~]$ qsub test.sh  
8.pbspro-server

> [test@pbspro-server ~]$ qsub test3.sh  
> 9.pbspro-server

1. the job status:

> [test@pbspro-server ~]$ qstat -a  
> pbspro-server: Req’d Req’d Elap  
> Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time  
> 8.pbspro-server opc workq pbs-openmp 14047 2 2 – – R 00:14  
> 9.pbspro-server opc workq test3.sh 14202 2 2 – – R 00:06

And there is exception in mom\_logs:

> 01/15/2019 13:11:28;0008;pbs\_mom;Job;8.pbspro-server;JOIN\_JOB as node 1  
> 01/15/2019 13:17:56;0001;pbs\_mom;Svr;pbs\_mom;Connection timed out (110) in open\_demux, open\_demux: connect 10.0.16.3:34297  
> 01/15/2019 13:17:56;0001;pbs\_mom;Job;8.pbspro-server;task not started, Failure orted -2  
> 01/15/2019 13:17:56;0008;pbs\_mom;Job;8.pbspro-server;no active tasks  
> 01/15/2019 13:19:35;0008;pbs\_mom;Job;9.pbspro-server;JOIN\_JOB as node 1  
> 01/15/2019 13:26:03;0001;pbs\_mom;Svr;pbs\_mom;Connection timed out (110) in open\_demux, open\_demux: connect 10.0.16.3:57329  
> 01/15/2019 13:26:03;0001;pbs\_mom;Job;9.pbspro-server;task not started, Failure orted -2  
> 01/15/2019 13:26:03;0008;pbs\_mom;Job;8.pbspro-server;no active tasks  
> 01/15/2019 13:26:03;0008;pbs\_mom;Job;9.pbspro-server;no active tasks

And “10.0.16.3” is one of the pbs worknodes. Have configured passwordless access from server to worknodes, worknodes to worknodes, worknodes to server.  
Could you please help check why my two jobs have hanged? Thanks a lot!

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [January 15, 2019, 2:05pm UTC](https://community.openpbs.org/t/openmpi-support/1396/7 "2019-01-15T14:05:11Z")

</div>

Could you please update below script :

> [@liuxiaopiao](#):
>
> [test@pbspro-server ~]$ cat test.sh  
> #!/bin/bash  
> #PBS -N pbs-openmpi-sh  
> #PBS -l select=2  
> #PBS -l place=scatter  
> cat PBS\_NODEFILE hostnumber=(cat $PBS\_NODEFILE | wc -l)  
> /opt/openmpi/1.10.7/bin/mpirun hostname

**To :**

#!/bin/bash  
#PBS -N pbs-openmpi-sh  
#PBS -l select=2:ncpus=2:mpiprocs=2  
#PBS -l place=scatter  
/opt/openmpi/1.10.7/bin/mpirun -np `cat $PBS_NODEFILE | wc -l` /bin/hostname

> [@liuxiaopiao](#):
>
> Could you please help check why my two jobs have hanged?

The issue is DNS / Name resolution

1. Check DNS ( forward and reverse resolution of pbs server , pbs nodes on the pbs server and from the pbs nodes )
2. Make sure /etc/hosts is populated with pbs server and compute node names/ips/alias and exist on all the nodes of the PBS Complex.
3. without using PBS Pro , can you run open-mpi by having a static hostfile of node names – please check this.

---

<div class="post-metadata">

**Author:** ![liuxiaopiao](https://avatars.discourse-cdn.com/v4/letter/l/65b543/32.png) [@liuxiaopiao](https://community.openpbs.org/u/liuxiaopiao)\
**Post date:** [January 15, 2019, 2:21pm UTC](https://community.openpbs.org/t/openmpi-support/1396/8 "2019-01-15T14:21:48Z")

</div>

Sure. Will try it following your guide. Thanks so much for the patience to help me.  
Still two questions:

1. I thought this may be due to the firewall inside the cluster.  
For pbs\_mom to pbs\_mom, which ports do we need to add firewall port allow list?

2. is the failure related to my openmpi installation file?

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [January 15, 2019, 2:28pm UTC](https://community.openpbs.org/t/openmpi-support/1396/9 "2019-01-15T14:28:58Z")

</div>

> [@liuxiaopiao](#):
>
> I thought this may be due to the firewall inside the cluster.  
> For pbs\_mom to pbs\_mom, which ports do we need to add firewall port allow list?

Ports : 15001 to 15004 and 17001

> [@liuxiaopiao](#):
>
> is the failure related to my openmpi installation file?

- open-mpi installation location should be accessible by all the nodes in the complex.
- /path/to/shared/openmpi400 should be accessible by all the nodes and users of that PBS Cluster
- firewall, selinux should be disabled
- stricthostkey checking disabled
- qsub -I -l select=2:ncpus=2:mpiprocs=2 ( first -I is - capital i for Ice , the second one is lower case l , l for love )  
- submit this interactive job  
- you will get a terminal , cat $PBS\_NODEFILE  
- with hostname list try ssh , check whether it works both ways

---

<div class="post-metadata">

**Author:** ![liuxiaopiao](https://avatars.discourse-cdn.com/v4/letter/l/65b543/32.png) [@liuxiaopiao](https://community.openpbs.org/u/liuxiaopiao)\
**Post date:** [January 16, 2019, 2:07am UTC](https://community.openpbs.org/t/openmpi-support/1396/10 "2019-01-16T02:07:44Z")

</div>

Thanks again for the info.  
Now the scripts are working by checking all the items you mentioned above.

Still have concerns for the firewall port allow list:  
I have saw the log info such like

> 01/15/2019 13:57:00;0001;pbs\_mom;Svr;pbs\_mom;Connection timed out (110) in open\_demux, open\_demux: connect 10.0.16.3:58649  
> 01/15/2019 14:17:55;0001;pbs\_mom;Svr;pbs\_mom;Connection timed out (110) in open\_demux, open\_demux: connect 10.0.16.3:45144

So what’s the port number such like 58649/45144 for? we would like to add some ports to the firewall allow list instead of disabling firewall. Could you please give us some suggestions?

Thanks a lot!

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [January 16, 2019, 8:24am UTC](https://community.openpbs.org/t/openmpi-support/1396/11 "2019-01-16T08:24:32Z")

</div>

For PBS the required ports are : 15001, 15002, 15003, 15004 , 15007, 17001  
Please refer: Table 4-1: Ports Used by PBS Daemons in TPP Mode from the below guide [https://www.pbsworks.com/pdfs/PBS18.2.3\_BigBook.pdf](https://www.pbsworks.com/pdfs/PBS18.2.3_BigBook.pdf) , regarding ports.

---

<div class="post-metadata">

**Author:** ![liuxiaopiao](https://avatars.discourse-cdn.com/v4/letter/l/65b543/32.png) [@liuxiaopiao](https://community.openpbs.org/u/liuxiaopiao)\
**Post date:** [January 16, 2019, 1:19pm UTC](https://community.openpbs.org/t/openmpi-support/1396/12 "2019-01-16T13:19:16Z")

</div>

> [@adarsh](#):
>
> Ports Used by PBS Daemons in TPP Mode

Thanks so much. No further questions. Very appreciate your help!