How to avoid passwordless ssh

Users/Site Administrators
#1

Hi experts.

I have installed openpbs-20.0.0 on one server and two execution nodes. (total 3 nodes)
I also checked that nodes are communicated well when creating vnodes via qmgr.

My question is

  1. Should I enable passwordless ssh?
    I want to avoid that a new user access to the execution node.

As I use NFS, I add two lines in ‘var/spool/pbs/mom_priv/config’:

$usecp *:/home /home
$usecp *:/data /data

(I found this in http://docs.adaptivecomputing.com/torque/3-0-5/6.2filesystems.php)

  1. With the setting I have done, openmpi does not work when I submit two-node jobs.

The output log shows that
‘ORTE was unable to reliably start one or more daemons. …’

I have checked that one-node job works well.
Is this due to the ssh authentication?

I set the zone to trusted with eno2 (internal network).

Thanks.

#2

  1. Yes please enable passwordless-SSH access for all the users of the system, with stricthost-key checking disabled or a way that suppresses this.

  2. Please compile openmpi with PBS Pro TM API , please follow the discussion here
    Openmpi support

  3. $usecp is used for stage in and stageout using cp command when locations are common to PBS Server and Compute Nodes

Just to be sure and to clarify: openpbs and torque are not the same. Please follow this link PBS Default number of nodes and number of cpus

By default if there is no tight integration between PBS Pro and OpenMPI then ssh or rsh launchers will be used https://www.open-mpi.org/faq/?category=running#pls-rsh-agent . If you have tight integration between PBS Pro and OpenMPI then pbs_tmrsh would be used transparently with full control on the spawned processes with accounting and process cleaning

Please check this guide: https://www.altair.com/pdfs/pbsworks/PBSAdminGuide19.2.3.pdf and go through the below sections:

  1. 10.14 How Processes are Started Using MPI and PBS
  2. 10.13.3.1 Compiling LAM MPI 7.x/Open MPI with the TM Module

Note:

  1. Static IP addresses / hostname / DNS (forward/reverse resolvable) - /etc/hosts populated on all the nodes with all the systems of the cluster (do not use DHCP).
  2. SELinux disabled (if disabled now , then system should be rebooted)
  3. passwordless-ssh for all the users
  4. Same UID and GID for all the users
  5. ports 15001 to 15009 and 17001 open between PBS server/compute nodes and between the nodes

Hope this helps

#3

Thanks for the reply.

I have downloaded open-mpi 4.0.4 on the website https://www.open-mpi.org/software/ompi/v4.0/ and tried to build it.

./configure --prefix=/opt/openmpi/4.0.4 --with-tm=/opt/pbs

In the configuration step, I cannot continue. Here is the error message:

— MCA component ess:tm (m4 configuration macro)
checking for MCA component ess:tm compile mode… static
checking --with-tm value… not found
configure: WARNING: Expected file /opt/pbs/include/tm.h not found
configure: error: Cannot continue

I downloaded rpm file of openpbs 20.0.0 and installed successfully.
How can I resolve it?

#4

UPDATE: Thanks to your note, I have checked that openpbs works well with host-based authentications (passwordless ssh).
And MPICH in centos 8 repository works well.
However, openmpi does not work with the error message on the first post.
There is no file in /opt/pbs/include/.

#5

@mkaro : could you please look into this. I can replicate this on CentOS 8 + openpbs 20.0.0 + openmpi-4.0.4.

  1. pbs-config file is missing
  2. the include directory is empty

Thank you

#6

You need to install the openpbs-devel package on the system you’re using to build OpenMPI. It contains the tm.h that OpenMPI is looking for.

#7

Thanks for reply.
openmpi-devel in centos 8 repository does not work. :frowning:
And I cannot find openpbs-devel.

I have downloaded openpbs 20.0.0 on the website, https://www.openpbs.org/Download.aspx#download
I have installed openpbs-server on the frontend node and openpbs-execution on the computing nodes. Both packages do not have ‘tm.h’ and ‘pbs-config’ files.

Could you tell me how to find and install openpbs-devel package?

#8

I generated the package for you. You can download it here:

https://altairengineering-my.sharepoint.com/:u:/g/personal/mkaro_altair_com/EZBwvuOfVoRGgmblYi7ErlcBxFCo7dG7KGSPoBmZ5nCyPA?e=uRgtf1

Link will expire at the end of the month. The next release will include this package.

1 Like
#9

Thank you @mkaro . I could compile openmpi4.0.4 after deploying the openpbs-devel .

#10

Thank you @adarsh @mkaro.
With openpbs-devel, I could compile openmpi 4.0.4 and it works well!

2 Likes