Configure OpenMPI with PBS Torque

[legio06]

Thank you for looking into this issue. I have a small cluster with a headnode and two compute nodes. All servers are running CentOS 7. I have been asked to install and configure Open MPI with PBS Torque. PBS Torque works fine we are able to submit jobs to the compute nodes.

Open MPI version is 4.0.5 and I have installed Open MPI from source.

• Operating system/version: CentOS 7
• Computer hardware: Headnode Intel - Compute nodes AMD
• Network type: TCP - NFS

Details of the problem

I have install and configure Open MPI with the following options “–prefix=/opt/openmpi-4.0.5 --with-tm --enable-orterun-prefix-by-default”. after the installation i ran “ompi_info | grep tm” on all the servers to make that “tm” was installed.

$ ompi_info |  grep tm
  Configure command line: '--prefix=/opt/openmpi-4.0.5' '--with-tm' '--enable-orterun-prefix-by-default'
                 MCA ess: tm (MCA v2.1.0, API v3.0.0, Component v4.0.5)
                 MCA plm: tm (MCA v2.1.0, API v2.0.0, Component v4.0.5)
                 MCA ras: tm (MCA v2.1.0, API v2.0.0, Component v4.0.5)

I can run simple “mpirun” test like the ones in the example of Open MPI directory.

$  mpirun --hostfile /opt/hostfile -np 64 --map-by node hostname

I even ran a “hello_c” program with an infinite loop in the code so i can “htop” on all the servers and see the allocated CPU’s.

I also ran the ring_c program which sends a simple MPI message around a ring 10 times.

$ mpirun  --hostfile /opt/hostfile -np 64 /opt/openmpi-4.0.5/examples/ring
Warning: Permanently added 'tq3,10.112.0.14' (ECDSA) to the list of known hosts.
Warning: Permanently added 'tq4,10.112.0.16' (ECDSA) to the list of known hosts.
Process 0 sending 10 to 1, tag 201 (64 processes in ring)
Process 0 sent to 1
Process 0 decremented value: 9
Process 0 decremented value: 8
Process 0 decremented value: 7
Process 0 decremented value: 6
Process 0 decremented value: 5
Process 0 decremented value: 4
Process 0 decremented value: 3
Process 0 decremented value: 2
Process 0 decremented value: 1
Process 0 decremented value: 0
Process 0 exiting
Process 1 exiting
etc..............

However, The issue i am having is when I submit “mpirun” job to PBS Torque only 1 core per server is utilized. Although, I specifically asked for all the cores in the cluster (64 cores - 32 cores per compute node).

Here is my shell script:

!/bin/bash
#PBS -k o
#PBS -l nodes=2:ppn=32,walltime=12:00:00
#PBS -W x=nmatchpolicy:exactnode
#PBS -N mpi_test_tq3_4

mpirun   /opt/openmpi-4.0.5/examples/ring

This is how i submit the job:

$ qsub -l nodes=2:ppn=32 -W x=nmatchpolicy:exactnode mpi_test.sh
49.mt-manager

The output file shows that only 2 processors were allocated.

$ cat  mpi_test_tq3_4.o49
Process 0 sending 10 to 1, tag 201 (2 processes in ring)
Process 0 sent to 1
Process 0 decremented value: 9
Process 0 decremented value: 8
Process 0 decremented value: 7
Process 0 decremented value: 6
Process 0 decremented value: 5
Process 0 decremented value: 4
Process 0 decremented value: 3
Process 0 decremented value: 2
Process 0 decremented value: 1
Process 0 decremented value: 0
Process 0 exiting
Process 1 exiting

We can see that only 1 processor was allocated to the job “Process 0 sending 10 to 1, tag 201 (2 processes in ring)”
Did I do something wrong when I configured Open MPI with the “–with-tm” ?
Should I have include the path to the PBS Torque install directory (–wiht-tm=/path/to/torque/directory) ?
Am i submitting the job with the proper $PBS option and “qsub” options ?

Thank you in advance for your help in this issue.

Eric

[adarsh]

openpbs and torque are different. Just in case this is PBS OSS / openpbs forum.

PBS Default number of nodes and number of cpus

So a little history for your knowledge. PBS was originally developed for NASA under a contract project. Altair Engineering acquired the rights to all the PBS technology and intellectual property from Veridian in 2003. Altair Engineering currently owns and maintains the intellectual property associated with PBS, and also employs the original development team from NASA. The following versions of PBS are currently available: OpenPBS — original open source version released by MRJ in 1998 (not ac…

Compiling OpenMPI with openpbs, this forum discussion might be helpful

Compile OpenMPI with PBSpro 14.1.10

In addition to the pbs-config and the LDFLAGS / LD_LIBRARY_PATH above, the error about “TM support requested but not found” can still occur. Looking in the config.log will show: configure:6266: gcc -L/opt/pbs/lib -lpbs -lpthread -lcrypto conftest.c >&5 /usr/bin/ld: cannot find -lcrypto collect2: error: ld returned 1 exit status The solution to this is to construct a symbolic link if libcrypto.so does not exist. (Thanks Alexis/Prasanna) Check the listing first: ls -al /usr/lib64/libcry…

Openmpi support

PBS Supports openmpi, but the openmpi should be compiled with PBS TM option. Thats correct, for this to work you need to compile openmpi with PBS TM , please follow the below

TM api on pbspro does not work for me?

fedele, We hope you have compiled OpenMPI from source as below: 1. tar -xvzf openmpi-XXX.tar.gz (current stable version as of now ) 2. export LIBS=-ldl 3. source /etc/pbs.conf 4. vi /opt/pbs/bin/pbs-config (chmod 755 ) copy paste the script mkaro has suggested in this link http://community.openpbs.org/t/compile-openmpi-with-pbspro-14-1-10/159/4 5. cd openmpi-XXX 6. ./configure --prefix=/appdata/openmpi/XXX --with-tm=/opt/pbs --enable-mpi-interface-warning --enable-shared --enable-static…

Thank you.

[legio06]

Thank you for the history course, I appreciate it. Always nice to learn new things.
I was actually researching on the internet a forum where users would the same issue that I do.
I came across OpenPBS and I thought that OpenPBS was “somewhat” similar as Torque and members could point me to the right direction which you did. I will look at the forum discussion you mentioned.

Thank you again for your help.