# Two networks in the cluster

**URL:** <https://community.openpbs.org/t/two-networks-in-the-cluster/4258>\
**Category:** Users/Site Administrators\
**Created:** [August 11, 2026, 11:33am UTC](https://community.openpbs.org/t/two-networks-in-the-cluster/4258 "2026-08-11T11:33:22Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![wakaka](https://avatars.discourse-cdn.com/v4/letter/w/d07c76/32.png) [@wakaka](https://community.openpbs.org/u/wakaka)\
**Post date:** [August 11, 2026, 11:33am UTC](https://community.openpbs.org/t/two-networks-in-the-cluster/4258/1 "2026-08-11T11:33:22Z")

</div>

The cluster has 25G and 100G networks. How should I configure it so that inter-node connections and communication use the 25G network, while cross-node parallel jobs are executed using the 100G network?

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [August 16, 2026, 12:18pm UTC](https://community.openpbs.org/t/two-networks-in-the-cluster/4258/2 "2026-08-16T12:18:59Z")

</div>

1. Give each node IP and hostname associated with 25G network  
192.25.0.1 headnoe-25g
2. Give each node IP and nostname associated with 100G network  
192.100.0.1 headnode-100g

Within the /etc/pbs.conf use the PBS\_SERVER name and PBS\_LEAF\_NAME appropriately.  
While running MPI jobs , you can re-construct the $PBS\_NODEFILE by rewriting into a separate file to check and replace the compute nodes mentioned in the $PBS\_NODEFILE file by 100G hostanmes.

Hope this helps

---

<div class="post-metadata">

**Author:** ![wakaka](https://avatars.discourse-cdn.com/v4/letter/w/d07c76/32.png) [@wakaka](https://community.openpbs.org/u/wakaka)\
**Post date:** [August 17, 2026, 1:39am UTC](https://community.openpbs.org/t/two-networks-in-the-cluster/4258/3 "2026-08-17T01:39:01Z")

</div>

You mean that in `/etc/pbs.conf`, the variable `PBS_SERVER` uses the hostname representing a 25G network, and the file `$PBS_NODEFILE` uses the hostname representing a 100G network. However, I have a question: the `$PBS_NODEFILE` file is automatically generated and defined after the job is submitted; it seems I cannot modify it, or rather, modifying it wouldn’t take effect.

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [August 17, 2026, 10:04am UTC](https://community.openpbs.org/t/two-networks-in-the-cluster/4258/4 "2026-08-17T10:04:09Z")

</div>

> [@wakaka](#):
>
> You mean that in `/etc/pbs.conf`, the variable `PBS_SERVER` uses the hostname representing a 25G network, and the file `$PBS_NODEFILE` uses the hostname representing a 100G network.

Yes, your understanding is correct.

> [@wakaka](#):
>
> However, I have a question: the `$PBS_NODEFILE` file is automatically generated and defined after the job is submitted; it seems I cannot modify it, or rather, modifying it wouldn’t take effect.

yes, it is generated dynamically by PBS, however you can read the file and create another update file as host.txt and reference it in your MPI batch command line

```auto
ls -ltr $PBS_NODEFILE
cat $PBS_NODEFILE
MPI_NODEFILE=host.txt
#update the hostname of the nodes in the PBS_NODEFILE, with their 100g hostnames 
sed 's/$/-100g/' "$PBS_NODEFILE" > "$MPI_NODEFILE" 
cat $MPI_NODEFILE
mpirun -np XX -hostfile $MPI_NODEFILE

```

---

<div class="post-metadata">

**Author:** ![wakaka](https://avatars.discourse-cdn.com/v4/letter/w/d07c76/32.png) [@wakaka](https://community.openpbs.org/u/wakaka)\
**Post date:** [August 17, 2026, 11:05am UTC](https://community.openpbs.org/t/two-networks-in-the-cluster/4258/5 "2026-08-17T11:05:59Z")

</div>

Is this the only way? There must be some programs that cannot use the hostfile or nodefile parameter.

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [August 20, 2026, 6:06am UTC](https://community.openpbs.org/t/two-networks-in-the-cluster/4258/6 "2026-08-20T06:06:07Z")

</div>

These are the options, if the applicaiton programs cannot use the hostfile or nodefile, then how are they identifying which compute nodes to run on. Please try checking pbsdsh (operates within the PBS allocation and uses PBS’s task-launching infrastructure.) option.