How to avoid passwordless ssh

  1. Yes please enable passwordless-SSH access for all the users of the system, with stricthost-key checking disabled or a way that suppresses this.

  2. Please compile openmpi with PBS Pro TM API , please follow the discussion here
    Openmpi support - #5 by adarsh

  3. $usecp is used for stage in and stageout using cp command when locations are common to PBS Server and Compute Nodes

Just to be sure and to clarify: openpbs and torque are not the same. Please follow this link PBS Default number of nodes and number of cpus - #4 by scott

By default if there is no tight integration between PBS Pro and OpenMPI then ssh or rsh launchers will be used FAQ: Running MPI jobs . If you have tight integration between PBS Pro and OpenMPI then pbs_tmrsh would be used transparently with full control on the spawned processes with accounting and process cleaning

Please check this guide: https://www.altair.com/pdfs/pbsworks/PBSAdminGuide19.2.3.pdf and go through the below sections:

  1. 10.14 How Processes are Started Using MPI and PBS
  2. 10.13.3.1 Compiling LAM MPI 7.x/Open MPI with the TM Module

Note:

  1. Static IP addresses / hostname / DNS (forward/reverse resolvable) - /etc/hosts populated on all the nodes with all the systems of the cluster (do not use DHCP).
  2. SELinux disabled (if disabled now , then system should be rebooted)
  3. passwordless-ssh for all the users
  4. Same UID and GID for all the users
  5. ports 15001 to 15009 and 17001 open between PBS server/compute nodes and between the nodes

Hope this helps