How to avoid passwordless ssh

Users/Site Administrators
1

Hi experts.

I have installed openpbs-20.0.0 on one server and two execution nodes. (total 3 nodes)
I also checked that nodes are communicated well when creating vnodes via qmgr.

My question is

  1. Should I enable passwordless ssh?
    I want to avoid that a new user access to the execution node.

As I use NFS, I add two lines in ‘var/spool/pbs/mom_priv/config’:

$usecp *:/home /home
$usecp *:/data /data

(I found this in http://docs.adaptivecomputing.com/torque/3-0-5/6.2filesystems.php)

  1. With the setting I have done, openmpi does not work when I submit two-node jobs.

The output log shows that
‘ORTE was unable to reliably start one or more daemons. …’

I have checked that one-node job works well.
Is this due to the ssh authentication?

I set the zone to trusted with eno2 (internal network).

Thanks.

2

  1. Yes please enable passwordless-SSH access for all the users of the system, with stricthost-key checking disabled or a way that suppresses this.

  2. Please compile openmpi with PBS Pro TM API , please follow the discussion here
    Openmpi support - #5 by adarsh

  3. $usecp is used for stage in and stageout using cp command when locations are common to PBS Server and Compute Nodes

Just to be sure and to clarify: openpbs and torque are not the same. Please follow this link PBS Default number of nodes and number of cpus - #4 by scott

By default if there is no tight integration between PBS Pro and OpenMPI then ssh or rsh launchers will be used FAQ: Running MPI jobs . If you have tight integration between PBS Pro and OpenMPI then pbs_tmrsh would be used transparently with full control on the spawned processes with accounting and process cleaning

Please check this guide: https://www.altair.com/pdfs/pbsworks/PBSAdminGuide19.2.3.pdf and go through the below sections:

  1. 10.14 How Processes are Started Using MPI and PBS
  2. 10.13.3.1 Compiling LAM MPI 7.x/Open MPI with the TM Module

Note:

  1. Static IP addresses / hostname / DNS (forward/reverse resolvable) - /etc/hosts populated on all the nodes with all the systems of the cluster (do not use DHCP).
  2. SELinux disabled (if disabled now , then system should be rebooted)
  3. passwordless-ssh for all the users
  4. Same UID and GID for all the users
  5. ports 15001 to 15009 and 17001 open between PBS server/compute nodes and between the nodes

Hope this helps

3

Thanks for the reply.

I have downloaded open-mpi 4.0.4 on the website https://www.open-mpi.org/software/ompi/v4.0/ and tried to build it.

./configure --prefix=/opt/openmpi/4.0.4 --with-tm=/opt/pbs

In the configuration step, I cannot continue. Here is the error message:

— MCA component ess:tm (m4 configuration macro)
checking for MCA component ess:tm compile mode… static
checking --with-tm value… not found
configure: WARNING: Expected file /opt/pbs/include/tm.h not found
configure: error: Cannot continue

I downloaded rpm file of openpbs 20.0.0 and installed successfully.
How can I resolve it?

4

UPDATE: Thanks to your note, I have checked that openpbs works well with host-based authentications (passwordless ssh).
And MPICH in centos 8 repository works well.
However, openmpi does not work with the error message on the first post.
There is no file in /opt/pbs/include/.

5

@mkaro : could you please look into this. I can replicate this on CentOS 8 + openpbs 20.0.0 + openmpi-4.0.4.

  1. pbs-config file is missing
  2. the include directory is empty

Thank you

6

You need to install the openpbs-devel package on the system you’re using to build OpenMPI. It contains the tm.h that OpenMPI is looking for.

7

Thanks for reply.
openmpi-devel in centos 8 repository does not work. :frowning:
And I cannot find openpbs-devel.

I have downloaded openpbs 20.0.0 on the website, https://www.openpbs.org/Download.aspx#download
I have installed openpbs-server on the frontend node and openpbs-execution on the computing nodes. Both packages do not have ‘tm.h’ and ‘pbs-config’ files.

Could you tell me how to find and install openpbs-devel package?

8

I generated the package for you. You can download it here:

https://altairengineering-my.sharepoint.com/:u:/g/personal/mkaro_altair_com/EZBwvuOfVoRGgmblYi7ErlcBxFCo7dG7KGSPoBmZ5nCyPA?e=uRgtf1

Link will expire at the end of the month. The next release will include this package.

1 Like
9

Thank you @mkaro . I could compile openmpi4.0.4 after deploying the openpbs-devel .

10

Thank you @adarsh @mkaro.
With openpbs-devel, I could compile openmpi 4.0.4 and it works well!

2 Likes