# Run OpenPBS in private network **URL:** \ **Category:** Users/Site Administrators\ **Created:** [April 18, 2024, 3:28pm UTC](https://community.openpbs.org/t/run-openpbs-in-private-network/3840 "2024-04-18T15:28:04Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![gurvirbawa](https://avatars.discourse-cdn.com/v4/letter/g/7bcc69/32.png) [@gurvirbawa](https://community.openpbs.org/u/gurvirbawa)\ **Post date:** [April 18, 2024, 3:28pm UTC](https://community.openpbs.org/t/run-openpbs-in-private-network/3840/1 "2024-04-18T15:28:04Z")
Hello, I have setup a small Linux cluster (1 server with 2 execution nodes) on university’s network (i.e., IPs 155.X.X.X). This setup works fine. We also have a private network for our department with IPs 10.X.X.X where the communication is much faster. So a user connects through university’s network through ssh abc@vortex.edu. Once in vortex.edu a user can only then connect to faster network through ssh vortex.data. Now I am trying to configure OpemPBS (23.06.06) such that it communicates over .data network instead of .edu network. So far, I have tried to change server name in /etc/pbs.conf to vortex.data on all the machines and while creating nodes, I entered node names as wx1.data and wx2.data. But I get below error messages: server\_logs: 04/18/2024 10:44:05;0c06;Server@vortex;TPP;Server@vortex(Thread 0);tpp\_mbox\_read;Unable to read from msg box 04/18/2024 10:44:05;0c06;Server@vortex;TPP;Server@vortex(Thread 0);tpp\_mbox\_read;Unable to read from msg box 04/18/2024 10:44:05;0001;Server@vortex;Svr;Server@vortex;is\_request, bad attempt to connect from 155.X.X.X:15003 mom\_logs: 04/18/2024 10:44:05;0c06;pbs\_mom;TPP;pbs\_mom(Thread 0);tpp\_mbox\_read;Unable to read from msg box 04/18/2024 10:44:05;0001;pbs\_mom;Svr;pbs\_mom;im\_eof, Premature end of message from addr 155.X.X.X:15001 on stream 110 04/18/2024 10:44:05;0002;pbs\_mom;Svr;im\_eof;Server closed connection. Any suggestions in this regard would be greatly appreciated. What settings should I check and change? (Just to mention again: OpenPBS works fine on cluster though network 155.X.X.X. I get these above messages when in pbsnodes I had mentioned wx1.data and wx2.data and in /etc/pbs.conf server name was vortex.data – not sure why in the error messages it still says bad request from 155.X.X.X) ---
**Author:** ![Source](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/source/32/756_2.png) [@Source](https://community.openpbs.org/u/Source)\ **Post date:** [April 19, 2024, 7:42am UTC](https://community.openpbs.org/t/run-openpbs-in-private-network/3840/2 "2024-04-19T07:42:27Z")
Dive into the PBS\_LEAF\_NAME or something? ---
**Author:** ![gurvirbawa](https://avatars.discourse-cdn.com/v4/letter/g/7bcc69/32.png) [@gurvirbawa](https://community.openpbs.org/u/gurvirbawa)\ **Post date:** [April 19, 2024, 12:14pm UTC](https://community.openpbs.org/t/run-openpbs-in-private-network/3840/3 "2024-04-19T12:14:11Z")
Thanks for the suggestion. I added PBS\_LEAF\_NAME to all /etc/pbs.conf i.e., on host server PBS\_LEAF\_NAME=vortex.data, and on two execution nodes PBS\_LEAF\_NAME is set to wx1.data and wx2.data respectively. Also, in /etc/hosts I’ve added the IPs and the names i.e., 10.X.X.X vortex.data, and so on two execution nodes. However, I get “qsub: Bad UID for job execution” when i submit the job now. Are there any other settings that need to be changed? ---
**Author:** ![gurvirbawa](https://avatars.discourse-cdn.com/v4/letter/g/7bcc69/32.png) [@gurvirbawa](https://community.openpbs.org/u/gurvirbawa)\ **Post date:** [April 19, 2024, 12:41pm UTC](https://community.openpbs.org/t/run-openpbs-in-private-network/3840/4 "2024-04-19T12:41:16Z")
Fixed Bad UID error by setting Qmgr: set server flatuid = True. But now the jobs stay in Q with the message: Insufficient amount of resource: ncpus (R: 2 A: 0 T: 0) I deleted nodes wx1 and wx2 and then added wx1.data and wx2.data. But it didn’t seem to make any difference. Below is the output of pbsnodes -av wxml1.data Mom = wxml1.data ntype = PBS state = state-unknown,down resources\_available.host = wxml1 resources\_available.vnode = wxml1.data resources\_assigned.accelerator\_memory = 0kb resources\_assigned.hbmem = 0kb resources\_assigned.mem = 0kb resources\_assigned.naccelerators = 0 resources\_assigned.ncpus = 0 resources\_assigned.vmem = 0kb resv\_enable = True sharing = default\_shared wxml2.data Mom = wxml2.data ntype = PBS state = state-unknown,down resources\_available.host = wxml2 resources\_available.vnode = wxml2.data resources\_assigned.accelerator\_memory = 0kb resources\_assigned.hbmem = 0kb resources\_assigned.mem = 0kb resources\_assigned.naccelerators = 0 resources\_assigned.ncpus = 0 resources\_assigned.vmem = 0kb resv\_enable = True sharing = default\_shared Any suggestions would be much appreciated. ---
**Author:** ![gurvirbawa](https://avatars.discourse-cdn.com/v4/letter/g/7bcc69/32.png) [@gurvirbawa](https://community.openpbs.org/u/gurvirbawa)\ **Post date:** [April 19, 2024, 1:06pm UTC](https://community.openpbs.org/t/run-openpbs-in-private-network/3840/5 "2024-04-19T13:06:51Z")
Als, comm\_logs are as below: 04/19/2024 08:27:32;0c06;Comm@vortex;TPP;Comm@vortex(Thread 2);tfd=16, Connection from leaf 10.1.0.48:15003 down 04/19/2024 08:27:37;0c06;Comm@vortex;TPP;Comm@vortex(Thread 1);tfd=14, Connection from leaf 10.1.0.58:15003 down SeELinux is disabled on these machines and I am able to connect to these via passwordless ssh. ---
**Author:** ![Source](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/source/32/756_2.png) [@Source](https://community.openpbs.org/u/Source)\ **Post date:** [April 19, 2024, 1:16pm UTC](https://community.openpbs.org/t/run-openpbs-in-private-network/3840/6 "2024-04-19T13:16:09Z")
From where comes the ‘m’ in the hostname? If there is no typo, check the hosts or DNS maybe? ---
**Author:** ![gurvirbawa](https://avatars.discourse-cdn.com/v4/letter/g/7bcc69/32.png) [@gurvirbawa](https://community.openpbs.org/u/gurvirbawa)\ **Post date:** [April 19, 2024, 1:18pm UTC](https://community.openpbs.org/t/run-openpbs-in-private-network/3840/7 "2024-04-19T13:18:06Z")
Sorry the names of execution nodes are wxml1 and wxml2. Or were you referring to any other typo? ---
**Author:** ![Source](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/source/32/756_2.png) [@Source](https://community.openpbs.org/u/Source)\ **Post date:** [April 19, 2024, 1:46pm UTC](https://community.openpbs.org/t/run-openpbs-in-private-network/3840/8 "2024-04-19T13:46:17Z")
I mean the ‘m’ in the nodenames exactly. It is said in the document we dont need to setup leaf for the pbs\_comm. Yet the error indicate it might be the problem. Try PBS\_LEAF\_ROUTERS from doc section 4.4.1 as well ```auto PBS_LEAF_ROUTERS=[:][,[:>port>]] ``` ---
**Author:** ![gurvirbawa](https://avatars.discourse-cdn.com/v4/letter/g/7bcc69/32.png) [@gurvirbawa](https://community.openpbs.org/u/gurvirbawa)\ **Post date:** [April 19, 2024, 2:05pm UTC](https://community.openpbs.org/t/run-openpbs-in-private-network/3840/10 "2024-04-19T14:05:36Z")
If I understood it correctly, does that mean add PBS\_LEAF\_ROUTERS on each execution node to point to vortex.data i.e., on both wxml1 and wxml2 add this value to /etc/pbs.conf: PBS\_LEAF\_ROUTERS=vortex.data (sorry if its a silly question) ---
**Author:** ![Source](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/source/32/756_2.png) [@Source](https://community.openpbs.org/u/Source)\ **Post date:** [April 20, 2024, 1:31pm UTC](https://community.openpbs.org/t/run-openpbs-in-private-network/3840/11 "2024-04-20T13:31:37Z")
Yah, setting on each execution node. I would suggest to add all the interfaces to PBS\_LEAF\_ROUTERS, such as PBS\_LEAF\_ROUTERS=vortex.data,vortex.edu ---
**Author:** ![gurvirbawa](https://avatars.discourse-cdn.com/v4/letter/g/7bcc69/32.png) [@gurvirbawa](https://community.openpbs.org/u/gurvirbawa)\ **Post date:** [April 20, 2024, 2:54pm UTC](https://community.openpbs.org/t/run-openpbs-in-private-network/3840/12 "2024-04-20T14:54:07Z")
Thanks for clarifying it. I made the changes but its not working. Perhaps details below would help. /etc/pbs.con (on hostnode) PBS\_EXEC=/opt/pbs PBS\_SERVER=vortex PBS\_LEAF\_NAME=vortex.data #PBS\_LEAF\_ROUTERS=vortex.data PBS\_START\_SERVER=1 PBS\_START\_SCHED=1 PBS\_START\_COMM=1 PBS\_START\_MOM=0 PBS\_HOME=/var/spool/pbs PBS\_CORE\_LIMIT=unlimited PBS\_SCP=/bin/scp /etc/pbs.info (for wxml1) PBS\_EXEC=/opt/pbs PBS\_SERVER=vortex PBS\_LEAF\_NAME=wxml1.data PBS\_START\_SERVER=0 PBS\_START\_SCHED=0 PBS\_START\_COMM=0 PBS\_START\_MOM=1 PBS\_HOME=/var/spool/pbs PBS\_CORE\_LIMIT=unlimited PBS\_SCP=/bin/scp PBS\_LEAF\_ROUTERS=vortex.data,vortex.dbwks.erau.edu /etc/pbs.conf (for wxml2) PBS\_EXEC=/opt/pbs PBS\_SERVER=vortex PBS\_LEAF\_NAME=wxml2.data PBS\_START\_SERVER=0 PBS\_START\_SCHED=0 PBS\_START\_COMM=0 PBS\_START\_MOM=1 PBS\_HOME=/var/spool/pbs PBS\_CORE\_LIMIT=unlimited PBS\_SCP=/bin/scp PBS\_LEAF\_ROUTERS=vortex.data,vortex.dbwks.erau.edu sched\_logs: 04/20/2024 10:39:45;0080;pbs\_sched;Req;;Starting Scheduling Cycle 04/20/2024 10:39:45;0080;pbs\_sched;Job;38.vortex;Considering job to run 04/20/2024 10:39:45;0040;pbs\_sched;Job;38.vortex;Insufficient amount of resource: ncpus (R: 2 A: 0 T: 0) 04/20/2024 10:39:45;0040;pbs\_sched;Job;38.vortex;Job will never run with the resources currently configured in the complex 04/20/2024 10:39:45;0080;pbs\_sched;Req;;Leaving Scheduling Cycle comm\_logs: 04/20/2024 10:50:14;0c06;Comm@vortex;TPP;Comm@vortex(Thread 3);tfd=18, Leaf registered address 10.1.0.48:15003 04/20/2024 10:50:14;0c06;Comm@vortex;TPP;Comm@vortex(Thread 3);tfd=18, Leaf 10.1.0.48:15003 still connected while another leaf connect arrived, dropping existing connection 16 04/20/2024 10:50:14;0c06;Comm@vortex;TPP;Comm@vortex(Thread 2);tfd=16, Connection from leaf 10.1.0.48:15003 down server\_logs: 04/20/2024 10:39:45;0100;Server@vortex;Req;;Type 0 request received from bawag@vortex.dbwks.erau.edu, sock=18 04/20/2024 10:39:45;0100;Server@vortex;Req;;Type 95 request received from bawag@vortex.dbwks.erau.edu, sock=19 04/20/2024 10:39:45;0100;Server@vortex;Req;;Type 21 request received from bawag@vortex.dbwks.erau.edu, sock=18 04/20/2024 10:39:45;0100;Server@vortex;Req;;Type 1 request received from bawag@vortex.dbwks.erau.edu, sock=18 04/20/2024 10:39:45;0100;Server@vortex;Req;;Type 3 request received from bawag@vortex.dbwks.erau.edu, sock=18 04/20/2024 10:39:45;0100;Server@vortex;Req;;Type 5 request received from bawag@vortex.dbwks.erau.edu, sock=18 04/20/2024 10:39:45;0100;Server@vortex;Job;38.vortex;enqueuing into gfs, state Q hop 1 04/20/2024 10:39:45;0008;Server@vortex;Job;38.vortex;Job Queued at request of bawag@vortex.dbwks.erau.edu, owner = bawag@vortex.dbwks.erau.edu, job name = pbsdshtest, queue = gfs 04/20/2024 10:39:45;0040;Server@vortex;Svr;vortex;Scheduler sent command 1 04/20/2024 10:39:45;0100;Server@vortex;Req;;Type 21 request received from Scheduler@vortex.dbwks.erau.edu, sock=16 04/20/2024 10:39:45;0100;Server@vortex;Req;;Type 71 request received from Scheduler@vortex.dbwks.erau.edu, sock=16 04/20/2024 10:39:45;0100;Server@vortex;Req;;Type 58 request received from Scheduler@vortex.dbwks.erau.edu, sock=16 04/20/2024 10:39:45;0100;Server@vortex;Req;;Type 20 request received from Scheduler@vortex.dbwks.erau.edu, sock=16 04/20/2024 10:39:45;0100;Server@vortex;Req;;Type 51 request received from Scheduler@vortex.dbwks.erau.edu, sock=16 04/20/2024 10:39:45;0100;Server@vortex;Req;;Type 96 request received from Scheduler@vortex.dbwks.erau.edu, sock=16 04/20/2024 10:39:45;0100;Server@vortex;Hook;;modifyjob event: accept req by default 04/20/2024 10:39:45;0008;Server@vortex;Job;38.vortex;Job Modified at request of Scheduler@vortex.dbwks.erau.edu mom\_logs (on wxml1) 04/20/2024 10:52:08;0c06;pbs\_mom;TPP;pbs\_mom(Thread 0);Registering address 10.1.0.48:15003 to pbs\_comm vortex.dbwks.erau.edu:17001 04/20/2024 10:52:08;0c06;pbs\_mom;TPP;pbs\_mom(Thread 0);Connected to pbs\_comm vortex.dbwks.erau.edu:17001 04/20/2024 10:52:08;0001;pbs\_mom;Svr;net\_restore\_handler;net restore handler called 04/20/2024 10:52:08;0c06;pbs\_mom;TPP;pbs\_mom(Thread 0);Connection to pbs\_comm vortex.dbwks.erau.edu:17001 down 04/20/2024 10:52:08;0001;pbs\_mom;Svr;net\_down\_handler;net down handler called 04/20/2024 10:52:08;0001;pbs\_mom;Svr;net\_restore\_handler;net restore handler called 04/20/2024 10:52:08;0c06;pbs\_mom;TPP;pbs\_mom(Thread 0);Connection to pbs\_comm vortex.data:17001 down mom\_logs (on wxml2): 04/20/2024 10:53:06;0c06;pbs\_mom;TPP;pbs\_mom(Thread 0);Registering address 10.1.0.58:15003 to pbs\_comm vortex.dbwks.erau.edu:17001 04/20/2024 10:53:06;0c06;pbs\_mom;TPP;pbs\_mom(Thread 0);Connected to pbs\_comm vortex.dbwks.erau.edu:17001 04/20/2024 10:53:06;0001;pbs\_mom;Svr;net\_restore\_handler;net restore handler called 04/20/2024 10:53:06;0c06;pbs\_mom;TPP;pbs\_mom(Thread 0);Connection to pbs\_comm vortex.dbwks.erau.edu:17001 down 04/20/2024 10:53:06;0001;pbs\_mom;Svr;net\_down\_handler;net down handler called 04/20/2024 10:53:06;0001;pbs\_mom;Svr;net\_restore\_handler;net restore handler called Am I doing something wrong? ---
**Author:** ![gurvirbawa](https://avatars.discourse-cdn.com/v4/letter/g/7bcc69/32.png) [@gurvirbawa](https://community.openpbs.org/u/gurvirbawa)\ **Post date:** [April 20, 2024, 3:05pm UTC](https://community.openpbs.org/t/run-openpbs-in-private-network/3840/13 "2024-04-20T15:05:55Z")
Also, below is the output from pbsnodes -av wxml1 Mom = wxml1.db.erau.edu ntype = PBS state = state-unknown,down pcpus = 1 resources\_available.host = wxml1 resources\_available.vnode = wxml1 resources\_assigned.accelerator\_memory = 0kb resources\_assigned.hbmem = 0kb resources\_assigned.mem = 0kb resources\_assigned.naccelerators = 0 resources\_assigned.ncpus = 0 resources\_assigned.vmem = 0kb resv\_enable = True sharing = default\_shared wxml2 Mom = wxml2.db.erau.edu ntype = PBS state = state-unknown,down pcpus = 1 resources\_available.host = wxml2 resources\_available.vnode = wxml2 resources\_assigned.accelerator\_memory = 0kb resources\_assigned.hbmem = 0kb resources\_assigned.mem = 0kb resources\_assigned.naccelerators = 0 resources\_assigned.ncpus = 0 resources\_assigned.vmem = 0kb resv\_enable = True sharing = default\_shared ---
**Author:** ![gurvirbawa](https://avatars.discourse-cdn.com/v4/letter/g/7bcc69/32.png) [@gurvirbawa](https://community.openpbs.org/u/gurvirbawa)\ **Post date:** [April 20, 2024, 5:58pm UTC](https://community.openpbs.org/t/run-openpbs-in-private-network/3840/14 "2024-04-20T17:58:35Z")
I think the issue is with comm, as per the error below: 04/20/2024 12:43:47;0c06;Comm@vortex;TPP;Comm@vortex(Thread 2);tfd=18, Leaf registered address 10.1.0.58:15003 04/20/2024 12:43:47;0c06;Comm@vortex;TPP;Comm@vortex(Thread 2);tfd=18, Leaf 10.1.0.58:15003 still connected while another leaf connect arrived, dropping existing connection 16 04/20/2024 12:43:47;0c06;Comm@vortex;TPP;Comm@vortex(Thread 1);tfd=16, Connection from leaf 10.1.0.58:15003 down 04/20/2024 12:43:47;0c06;Comm@vortex;TPP;Comm@vortex(Thread 1);tfd=18, Leaf registered address 10.1.0.48:15003 04/20/2024 12:43:47;0c06;Comm@vortex;TPP;Comm@vortex(Thread 1);tfd=18, Leaf 10.1.0.48:15003 still connected while another leaf connect arrived, dropping existing connection 16 04/20/2024 12:43:47;0c06;Comm@vortex;TPP;Comm@vortex(Thread 3);tfd=16, Connection from leaf 10.1.0.48:15003 down Any suggestions would be highly appreciated. ---
**Author:** ![gurvirbawa](https://avatars.discourse-cdn.com/v4/letter/g/7bcc69/32.png) [@gurvirbawa](https://community.openpbs.org/u/gurvirbawa)\ **Post date:** [April 21, 2024, 9:04pm UTC](https://community.openpbs.org/t/run-openpbs-in-private-network/3840/15 "2024-04-21T21:04:51Z")
Fixed! Followed steps from: > [@Proper way to configure PBS on multiple NIC system](https://community.openpbs.org/t/proper-way-to-configure-pbs-on-multiple-nic-system/1508): > > I’ve always been confused with all the hostname stuffs, and today I determined to dig into part of the source code after being tired about endless tries. I suppose it’s common to see a cluster running PBS, where the master node has two NICs, one for external netwrok, and one for intra-cluster communication. Let’s say, the cluster was named “HPC”. So it’s reasonable to configure the hosts and other files as follows: #hosts 127.0.0.1 localhost 1.2.3.4 HPC.your.domain.name HPC 10.… Created nodes using leaf names i.e., wxml1.data and wxml2.data. Added PBS-LEAF\_ROUTERS=vortex.data attribute to each MOM.