- @mkaro suggestion - recommended
You can run openmpi with PBS Pro without compiling with PBS TM option, but it will be loose integration and the communication and process launch will be via ssh and upon job deletion (job clean up is not correct) , zombie process might be created which needs to be cleaned up manually (Otherwise, pbs_attach would also help ) and usage data respect to each of the processes is not accounted.
If you use the MPI compiled with PBS TM , then the spawned processes are tightly controlled by PBS mom, proper accounting of the usage and job cleanup is guarantied.
How to compile?
#wget https://download.open-mpi.org/release/open-mpi/v4.0/openmpi-4.0.0.tar.bz2
#bzip2 -d openmpi-4.0.0.tar.bz2
#tar -xvf openmpi-4.0.0.tar
#cd openmpi-4.0.0
#export LD_LIBRARY_PATH=/opt/pbs/lib:$LD_LIBRARY_PATH
#export LDFLAGS=“-L/opt/pbs/lib -lpbs -lpthread -lcrypto”
#./configure --prefix=/path/to/shared/openmpi400 --with-tm=/opt/pbs/default --enable-mpi-interface-warning --enable-shared --enable-static --enable-cxx-exceptions
#make; make install
How to run ?
#!/bin/bash
#PBS -N pbs-openmpi-sh
#PBS -l select=2:ncpus=4:mpiprocs=4
#PBS -l place=scatter
/opt/openmpi400/bin/mpirun -np cat $PBS_NODEFILE | wc -l /bin/hostname
qsub pbs-openmpi.sh