The cluster has 25G and 100G networks. How should I configure it so that inter-node connections and communication use the 25G network, while cross-node parallel jobs are executed using the 100G network?
- Give each node IP and hostname associated with 25G network
192.25.0.1 headnoe-25g - Give each node IP and nostname associated with 100G network
192.100.0.1 headnode-100g
Within the /etc/pbs.conf use the PBS_SERVER name and PBS_LEAF_NAME appropriately.
While running MPI jobs , you can re-construct the $PBS_NODEFILE by rewriting into a separate file to check and replace the compute nodes mentioned in the $PBS_NODEFILE file by 100G hostanmes.
Hope this helps
You mean that in /etc/pbs.conf, the variable PBS_SERVER uses the hostname representing a 25G network, and the file $PBS_NODEFILE uses the hostname representing a 100G network. However, I have a question: the $PBS_NODEFILE file is automatically generated and defined after the job is submitted; it seems I cannot modify it, or rather, modifying it wouldn’t take effect.
Yes, your understanding is correct.
yes, it is generated dynamically by PBS, however you can read the file and create another update file as host.txt and reference it in your MPI batch command line
ls -ltr $PBS_NODEFILE
cat $PBS_NODEFILE
MPI_NODEFILE=host.txt
#update the hostname of the nodes in the PBS_NODEFILE, with their 100g hostnames
sed 's/$/-100g/' "$PBS_NODEFILE" > "$MPI_NODEFILE"
cat $MPI_NODEFILE
mpirun -np XX -hostfile $MPI_NODEFILE
Is this the only way? There must be some programs that cannot use the hostfile or nodefile parameter.
These are the options, if the applicaiton programs cannot use the hostfile or nodefile, then how are they identifying which compute nodes to run on. Please try checking pbsdsh (operates within the PBS allocation and uses PBS’s task-launching infrastructure.) option.