Lumerical mpi syntax mismatch with pbs mpi

Users/Site Administrators
1

Need your inputs:

pasting mpi lines from below link

General MPI Syntax

mpiexec [mpi_options] solver [solver_options]

MPI Options

-hosts

  • hostfile : text file with 1 hostname/IP per line, with corresponding number of processes separated by a comma ':'

we observed:

pbs hostfile is as below

node12
node12
node7

it will not work as lumerical mpi syntax

/home/apps/lumerical/v211/mpich2/nemesis/bin/mpiexec -hosts $PBS_NODEFILE /home/apps/lumerical/v211/bin/fdtd-engine-mpich2nem -t 1 -logall $PBS_O_WORKDIR/test.fsp

Can something be done here ?

PBS script:

#!/bin/sh
#PBS -N testmpi
#PBS -l select=2:ncpus=32:mpiprocs=32:mem=64gb
#PBS -q test
SIMFILE=mwp.fsp
LOG=mwp.log
cd $PBS_O_WORKDIR
echo $PBS_O_WORKDIR
cat $PBS_NODEFILE
/home/apps/lumerical/v211/mpich2/nemesis/bin/mpiexec hostname
/home/apps/lumerical/v211/mpich2/nemesis/bin/mpiexec -hosts $PBS_NODEFILE /home/apps/lumerical/v211/bin/fdtd-engine-mpich2nem -t 1 -logall $PBS_O_WORKDIR/test.fsp

2

Which MPI flavor are you using (MPICH, OpenMPI, etc.)? Was it compiled with task manager (TM) support for PBS/Torque? If not, it’s likely attempting to use ssh to spawn tasks on the nodes in $PBS_NODEFILE. Depending on the configuration of your nodes, this will likely fail.

3

You would have to read the $PBS_NODEFILE and then write into new file to your required format & use that file or create a script that produces output in the required format by the MPI. As @mkaro mentioned, if you have compiled the flavour of MPI with PBS task manager ™ , then the $PBS_NODEFILE would be automatically picked up by PBS without requiring to specify the -hosts option.

cat $PBS_NODEFILE | sort | uniq -c | awk ‘{print ($2":"$1)};’| xargs | sed -e ‘s/ /,/g’

4

@mkaro @adarsh its helpful pointers. thank you.