Compile OpenMPI with PBSpro 14.1.10

Attempting to compile OpenMPI for use with my cluster running PBSPro 14.1.0 and receiving the following error from the configure command:

— MCA component ess:tm (m4 configuration macro)
checking for MCA component ess:tm compile mode… static
checking --with-tm value… sanity check ok (/opt/pbs)
checking for pbs-config… not found
checking tm.h usability… yes
checking tm.h presence… yes
checking for tm.h… yes
looking for library in lib
checking for library containing tm_init… no
looking for library in lib64
checking for library containing tm_init… no
looking for library in lib
checking for library containing tm_init… no
looking for library in lib64
checking for library containing tm_init… no
configure: error: TM support requested but not found. Aborting

Excerpt from config.log:

configure:133689: checking for library containing tm_init
configure:133720: gcc -std=gnu99 -o conftest -O3 -DNDEBUG -finline-functions -fno-strict-aliasing -pthread -I/usr/local/src/openmpi-1.10.1/opal/mca/hwloc/hwloc191/hwloc/include -I/usr/local/src/openmpi-1.10.1/opal/mca/event/libevent2021/libevent -I/usr/local/src/openmpi-1.10.1/opal/mca/event/libevent2021/libevent/include -I/opt/pbs/include -L/opt/pbs/lib conftest.c -lrt -lm -lutil >&5
/tmp/ccPqRZ2s.o: In function main':** **conftest.c:(.text.startup+0x7): undefined reference totm_init’
collect2: error: ld returned 1 exit status
configure:133720: $? = 1

My configure line is as follows:
./configure --prefix=/usr/local/software/OpenMPI/ --with-tm=/opt/pbs --with-verbs

The /opt/pbs/include does have the tm.h file

Anyone have an idea as to what I’m missing here?

-SS-

Forgot to include that I’m attempting this on a CentOS 7.2 server.

That seems odd. Here’s what I’m seeing:

$ nm -D /opt/pbs/lib/libpbs.so | grep tm_init
00000000000398d0 T tm_init

It looks as though the exported symbol is present in the shared library. Do you see the same output on your system?

What version of OpenMPI are you using?

Looking at the call to the compiler in the config.log you will notice that “-lpbs” is not one of the arguments provided. It looks like the method used to check for tm_init() defined in config/orte_check_tm.m4 relies on a script called pbs-config being present. It uses this script to gather information like CPPFLAGS, LDFLAGS, and LIBS that are required for linking the library. I think the easiest approach would be to create a new version of pbs-config for PBS Pro, rather than attempt to change OpenMPI.

Are you able to file a ticket in our JIRA system? It is located here: https://pbspro.atlassian.net/secure/Dashboard.jspa

This script should get you past the problem you are seeing. You should be able to install it anywhere in your path. I installed it as /opt/pbs/bin/pbs-config for my testing. Remember to make it executable.

#!/bin/sh
#
#  Copyright (C) 1994-2016 Altair Engineering, Inc.
#  For more information, contact Altair at www.altair.com.
#
#  This file is part of the PBS Professional ("PBS Pro") software.
#
#  Open Source License Information:
#
#  PBS Pro is free software. You can redistribute it and/or modify it under the
#  terms of the GNU Affero General Public License as published by the Free
#  Software Foundation, either version 3 of the License, or (at your option) any
#  later version.
#
#  PBS Pro is distributed in the hope that it will be useful, but WITHOUT ANY
#  WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
#  PARTICULAR PURPOSE.  See the GNU Affero General Public License for more details.
#
#  You should have received a copy of the GNU Affero General Public License along
#  with this program.  If not, see <http://www.gnu.org/licenses/>.
#
#  Commercial License Information:
#
#  The PBS Pro software is licensed under the terms of the GNU Affero General
#  Public License agreement ("AGPL"), except where a separate commercial license
#  agreement for PBS Pro version 14 or later has been executed in writing with Altair.
#
#  Altair’s dual-license business model allows companies, individuals, and
#  organizations to create proprietary derivative works of PBS Pro and distribute
#  them - whether embedded or bundled with other software - under a commercial
#  license agreement.
#
#  Use of Altair’s trademarks, including but not limited to "PBS™",
#  "PBS Professional®", and "PBS Pro™" and Altair’s logos is subject to Altair's
#  trademark licensing policies.

. ${PBS_CONF_FILE:-/etc/pbs.conf}

case "$1" in
--prefix)
        [ -d "$PBS_EXEC" ] || exit 1
        echo $PBS_EXEC
        ;;
--cflags)
        [ -d "$PBS_EXEC/include" ] || exit 1
        echo "-I$PBS_EXEC/include"
        ;;
--libs)
        libdir="$PBS_EXEC/lib"
        [ -d "$libdir" ] || libdir="$PBS_EXEC/lib64"
        [ -d "$libdir" ] || exit 1
        echo "-L$libdir -lpbs -lpthread -lcrypto"
        ;;
*)
        exit 1
        ;;
esac

Filed JIRA ticket PP-360.

https://pbspro.atlassian.net/browse/PP-360

Michael,
Thank you for the quick analysis and script to try and address the issue. I will be attempting the recompile later. One question though is why is there only a /opt/pbs/lib directory, but not a /opt/pbs/lib64 which you would expect to find with x86_64 compiles?

-SS-

1 Like

Script worked. I was able to successfully compile OpenMPI 2.0.0.

1 Like

WRT lib64, it depends on the Linux distro. On CentOS it uses lib by default, but on openSUSE it uses lib64. I don’t know why that is the case, but it has been my observation. We could force it to use one or the other by specifying parameters to the configure script, but we have chosen not to. It could cause problems if someone ever tried to port PBS Pro to a non-Linux OS.

Thank you for the followup.

Adding a pbs_config script is one way of making openmpi’s configure to succeed, but here also a simple manual way to do it:

export LD_LIBRARY_PATH=/opt/pbs/lib:$LD_LIBRARY_PATH
export LDFLAGS="-L/opt/pbs/lib -lpbs -lpthread -lcrypto"
./configure --with-tm=/opt/pbs

This should simply configure openmpi with PBS tm support. I tried with 2.0.2 as well.

Hi mkaro,

  1. I have tried to compile OpenMpi (v1.10.6, v2.0.2, v2.1.0) but
    keep on encountering following error:
    “configure: error: TM support requested but not found. Aborting”

    pbspro: pbspro-server-14.1.0-13.1.x86_64 (install via rpm)
    openmpi: openmpi-2.1.0.tar.gz
    OS: Centos 7.1.1503

  2. 1st attempt:

    $ cd ~/openmpi-2.1.0
    $ ./configure --prefix=/opt1/openmpi-2.1.0 --with-tm=/opt/pbs

    — MCA component ess:tm (m4 configuration macro)
    checking for MCA component ess:tm compile mode… dso
    checking --with-tm value… sanity check ok (/opt/pbs)
    checking for pbs-config… not found <-- *** note this!!
    checking tm.h usability… yes
    checking tm.h presence… yes
    checking for tm.h… yes
    looking for library in lib
    checking for library containing tm_init… no
    looking for library in lib64
    checking for library containing tm_init… no
    looking for library in lib
    checking for library containing tm_init… no
    looking for library in lib64
    checking for library containing tm_init… no
    configure: error: TM support requested but not found. Aborting

    <-- initial attempt failed!

  3. 2nd attempt:

    Ref:
    Compile OpenMPI with PBSpro 14.1.10

    <-- confirmed that the ‘tm_init’ is in ‘libpbs.so’ library
    $ nm -D /opt/pbs/lib/libpbs.so | grep tm_init
    00000000000397e0 T tm_init

    Follow “mkaro solution” to create “/opt/pbs/bin/pbs-config” script,
    then re-config the openmpi again

    $ cd ~/openmpi-2.1.0
    $ ./configure --prefix=/opt1/openmpi-2.1.0 --with-tm=/opt/pbs

    — MCA component ess:tm (m4 configuration macro)
    checking for MCA component ess:tm compile mode… dso
    checking --with-tm value… sanity check ok (/opt/pbs)
    checking for pbs-config… /opt/pbs/bin/pbs-config <-- *** note this!!
    checking tm.h usability… yes
    checking tm.h presence… yes
    checking for tm.h… yes
    checking for tm_finalize… no
    checking tm.h usability… yes
    checking tm.h presence… yes
    checking for tm.h… yes
    looking for library in lib
    checking for library containing tm_init… no
    looking for library in lib64
    checking for library containing tm_init… no
    looking for library in lib
    checking for library containing tm_init… no
    looking for library in lib64
    checking for library containing tm_init… no
    configure: error: TM support requested but not found. Aborting

    $ vi config.log

    configure:161051: result: /opt/pbs/bin/pbs-config
    configure:161064: orte_check_tm_CPPFLAGS from pbs-config: -I/opt/pbs/include
    configure:161145: orte_check_tm_LDFLAGS from pbs-config: -L/opt/pbs/lib
    configure:161226: orte_check_tm_LIBS from pbs-config: -lpbs -lpthread -lcrypto
    configure:161241: checking tm.h usability


    configure:161553: checking for library containing tm_init
    configure:161584:
    gcc -std=gnu99 -o conftest -O3 -DNDEBUG -finline-functions
    -fno-strict-aliasing -pthread
    -I/home/richardl/openmpi-2.1.0/opal/mca/hwloc/hwloc1112/hwloc/include
    -I/home/richardl/openmpi-2.1.0/opal/mca/event/libevent2022/libevent
    -I/home/richardl/openmpi-2.1.0/opal/mca/event/libevent2022/libevent/include
    -I/opt/pbs/include
    -L/opt/pbs/lib conftest.c -lrt -lm -lutil >&5/tmp/cc4fiK8N.o:
    In function main': conftest.c:(.text.startup+0x7): undefined reference totm_init’
    collect2: error: ld returned 1 exit status
    configure:161584: $? = 1

    <-- the script “/opt/pbs/bin/pbs-config” did execute properly
    <-- Looking at the call to the compiler in the config.log will
    notice that “-lpbs” is still not one of the arguments provided

    <-- So, your solution of creating "/opt/pbs/bin/pbs-config"
    did not solve the problem!!

    Is it the problem with the pbspro-14.1.0 rpm package? Any idea?
    Hope that you can point me to the right direction!

Rgds,
Richardl.

In addition to the pbs-config and the LDFLAGS / LD_LIBRARY_PATH above, the error about “TM support requested but not found” can still occur.

Looking in the config.log will show:

configure:6266: gcc -L/opt/pbs/lib -lpbs -lpthread -lcrypto conftest.c >&5
/usr/bin/ld: cannot find -lcrypto
collect2: error: ld returned 1 exit status

The solution to this is to construct a symbolic link if libcrypto.so does not exist. (Thanks Alexis/Prasanna)
Check the listing first:
ls -al /usr/lib64/libcrypto*
19 May 12 13:43 /usr/lib64/libcrypto.so → libcrypto.so.1.0.1e
1440448 Jun 24 2014 /usr/lib64/libcrypto.so.0.9.8e
19 Aug 20 2016 /usr/lib64/libcrypto.so.10 → libcrypto.so.1.0.1e
2017168 May 9 2016 /usr/lib64/libcrypto.so.1.0.1e
19 May 27 2015 /usr/lib64/libcrypto.so.6 → libcrypto.so.0.9.8e

In ‘this’ case, create a symbolic link:
cd /usr/lib64
ln -s libcrypto.so.1.0.1e libcrypto.so

Then configure should work, followed by the remaining process:
./configure --with-tm=/opt/pbs
make
make install

To Compile an openMPI program:
mpiCC -g wave_mpi.cpp -o wave_mpi

Compiling a previously working MPI program will FAIL with the following:
… error: ‘MPI’ has not been declared

The reason is that a change from OpenMPI 1.x to 2.x correspond roughly to change in impementation of MPI v2 to MPI v3

Apparently MPI-3.x deleted C++ bindings so OpenMPI is not build by default with C++

Solution is:
make clean
make distclean
./configure --with-tm=/opt/pbs --enable-mpi-cxx
make
make install

Then to compile program:
mpiCC -g wave_mpi.cpp -o wave_mpi

THis can be submitted using a script:

(replace HASH_ with symbol #)

Submitting to pbs:
qsub wave_pbs.sh

hi subhasisb,
I can successfully install openmpi according to your method, but because openmpi depends on libpbs.so.0(), you need to install the provider of libpbs.so.0, that is, the RPM package of PBS, before installing openmpi

Is there any way to decouple openmpi from PBS?

I try to link the static library, that is, openmpi does not rely on libpbs.so.0, but directly links libpbs. a, as follows:

export LDFLAGS="-L/opt/pbs/lib -Wl,-Bstatic -lpbs -Wl,-Bdynamic -lpthread -lcrypto"

But it didn’t work. openmpi still need libpbs.so when install.
Do you have any suggestions? Thank you!

After installing the openpbs-devel package you should check for the presence of the /opt/pbs/lib/libpbs.a file. This is the archive library needed to link OpenPBS statically against OpenMPI. Assuming the file is present, try the following LDFLAGS setting:

export LDFLAGS="/opt/pbs/lib/libpbs.a -lpthread -lcrypto"

Thanks for your reply. I’ve tried the method you mentioned, but openmpi still relies on libpbs.so in some places. I’m not familiar with its configuration and source code, so I’m confused

My attempts are as follows:

openmpi-4.1.1.tar.gz + openpbs 19.1.3, centOS7.8

Because openpbs v19.1.3 does not provide PBS devel package, I directly installed pbsserver package to extract /opt/pbs/include and /opt/pbs/lib, which act as pbs-devel.rpm. Then remove pbs rpm,just use include and lib.

First of all, I used export LDFLAGS= “- L /opt/pbs/lib -lpbs - lpthread - lcrypto” to successfully compile and create the RPM package of openmpi, which depends on libpbs.so.0. However, when installing openmpi.rpm, I must install pbs.rpm which is provider of libpbs.so.0, then I can install openmpi.rpm, just as shown in my topic.

Later, I tried the export LDFLAGS=’/opt/pbs/lib/libpbs. a -lpthread -lcrypto’ as you mentioned above, and deleted /opt/pbs/lib/libpbs.so to prevent confusion. However, when compiling openmpi.rpm, some place also need libpbs.so. I think this may be because --with-tm=/opt/pbs, so some one just use /opt/pbs/lib/libpbs.so.

Then, I tried to export LIBS=’/opt/pbs/lib/ libpbs. a - lpthread - lcrypto’ instead of LDFLAGS. It seems that openmpi no longer depends on libpbs.so, but its source code and libpbs.a’s interface tm_initxxx multi defined,error.

Q1: Can you help me reproduce and locate the problem according to the steps I described?

Q2:In addition, if the default export LDFLAGS=’- L/opt/pbs/lib -lpbs - lpthread - lcrypto’ is adopted, that is, it depends on libpbs.so.0. Therefore, I need install openpbs-server-xxx.rpm before installing openmpi.rpm. Is this a standardized process?
I think even if openpbs 20.0.1 is used and pbs-devel.rpm is used during openmpi.rpm’s compilation, the installation of openmpi.rpm will still rely on the RPM of PBS.

thank you!!!

Please refer to an earlier post in this thread: Compile OpenMPI with PBSpro 14.1.10 - #5 by mkaro

You’ll need to customize the script so that the --libs parameter returns “/opt/pbs/lib/libpbs.a -lpthread -lcrypto”

The autotools macros used to detect PBS are not very sophisticated and rely on this script being present. If you look for files in the OpenMPI tree that end with check_tm.m4 you’ll see how they rely on the script being present.

The only other option I see would be to update the macros in the OpenMPI tree and (hopefully) contribute them back to the community.