Thank you @adarsh. I opened those ports and restarted PBS, but no luck yet. Here is the /var/spool/pbs/server_logs/20241211 file:
cat /var/spool/pbs/server_logs/20241211
12/11/2024 09:14:32;0002;Server@ubuntu;Svr;Log;Log opened
12/11/2024 09:14:32;0002;Server@ubuntu;Svr;Server@ubuntu;pbs_version=23.06.06
12/11/2024 09:14:32;0002;Server@ubuntu;Svr;Server@ubuntu;pbs_build=mach=N/A:security=N/A:configure_args=N/A
12/11/2024 09:14:32;0002;Server@ubuntu;Svr;Server@ubuntu;hostname=ubuntu;pbs_leaf_name=N/A;pbs_mom_node_name=N/A
12/11/2024 09:14:32;0002;Server@ubuntu;Svr;Server@ubuntu;ipv4 interface lo: localhost
12/11/2024 09:14:32;0002;Server@ubuntu;Svr;Server@ubuntu;ipv4 interface enp5s0: ubuntu
12/11/2024 09:14:32;0002;Server@ubuntu;Svr;Server@ubuntu;ipv6 interface lo: localhost
12/11/2024 09:14:32;0002;Server@ubuntu;Svr;Server@ubuntu;ipv6 interface enp5s0: ubuntu
12/11/2024 09:14:32;0006;Server@ubuntu;Fil;Server@ubuntu;Version 23.06.06, started, initialization type = 1
12/11/2024 09:14:32;0002;Server@ubuntu;Svr;Server@ubuntu;pbs_status_db exit code 1
12/11/2024 09:14:32;0002;Server@ubuntu;Svr;Server@ubuntu;Starting PBS dataservice
12/11/2024 09:14:35;0002;Server@ubuntu;Svr;Server@ubuntu;connected to PBS dataservice@ubuntu
12/11/2024 09:14:35;0d80;Server@ubuntu;TPP;Server@ubuntu(Main Thread);TPP authentication method = resvport
12/11/2024 09:14:35;0c06;Server@ubuntu;TPP;Server@ubuntu(Main Thread);TPP leaf node names = 172.16.4.1:15001,127.0.0.1:15001,172.16.4.1:15001
12/11/2024 09:14:35;0d80;Server@ubuntu;TPP;Server@ubuntu(Main Thread);Initializing TPP transport Layer
12/11/2024 09:14:35;0d80;Server@ubuntu;TPP;Server@ubuntu(Main Thread);Max files allowed = 16384
12/11/2024 09:14:35;0d80;Server@ubuntu;TPP;Server@ubuntu(Main Thread);TPP initialization done
12/11/2024 09:14:35;0d80;Server@ubuntu;TPP;Server@ubuntu(Main Thread);Connecting to pbs_comm ubuntu:17001
12/11/2024 09:14:35;0c06;Server@ubuntu;TPP;Server@ubuntu(Thread 0);Thread ready
12/11/2024 09:14:35;0002;Server@ubuntu;n/a;setup_env;read environment from /var/spool/pbs/pbs_environment
12/11/2024 09:14:35;0000;Server@ubuntu;Svr;Server@ubuntu;Supported authentication method: resvport
12/11/2024 09:14:35;0c06;Server@ubuntu;TPP;Server@ubuntu(Thread 0);tpp_mbox_read;Unable to read from msg box
12/11/2024 09:14:35;0c06;Server@ubuntu;TPP;Server@ubuntu(Thread 0);tpp_mbox_read;Unable to read from msg box
12/11/2024 09:14:35;0c06;Server@ubuntu;TPP;Server@ubuntu(Thread 0);tpp_mbox_read;Unable to read from msg box
12/11/2024 09:14:35;0c06;Server@ubuntu;TPP;Server@ubuntu(Thread 0);Registering address 172.16.4.1:15001 to pbs_comm ubuntu:17001
12/11/2024 09:14:35;0c06;Server@ubuntu;TPP;Server@ubuntu(Thread 0);Connected to pbs_comm ubuntu:17001
12/11/2024 09:14:35;0c06;Server@ubuntu;TPP;Server@ubuntu(Thread 0);tpp_mbox_read;Unable to read from msg box
12/11/2024 09:14:35;0c06;Server@ubuntu;TPP;Server@ubuntu(Thread 0);tpp_mbox_read;Unable to read from msg box
12/11/2024 09:14:35;0002;Server@ubuntu;Svr;Server@ubuntu;Stopping PBS dataservice
12/11/2024 09:14:36;0c06;Server@ubuntu;TPP;Server@ubuntu(Thread 0);tpp_mbox_read;Unable to read from msg box
12/11/2024 09:14:36;0c06;Server@ubuntu;TPP;Server@ubuntu(Thread 0);tpp_mbox_read;Unable to read from msg box
And here is the /var/spool/pbs/comm_logs/20241211 file:
12/11/2024 09:14:32;0002;Comm@ubuntu;Svr;Log;Log opened
12/11/2024 09:14:32;0002;Comm@ubuntu;Svr;Comm@ubuntu;pbs_version=23.06.06
12/11/2024 09:14:32;0002;Comm@ubuntu;Svr;Comm@ubuntu;pbs_build=mach=N/A:security=N/A:configure_args=N/A
12/11/2024 09:14:32;0002;Comm@ubuntu;Svr;Comm@ubuntu;hostname=ubuntu;pbs_leaf_name=N/A;pbs_mom_node_name=N/A
12/11/2024 09:14:32;0002;Comm@ubuntu;Svr;Comm@ubuntu;ipv4 interface lo: localhost
12/11/2024 09:14:32;0002;Comm@ubuntu;Svr;Comm@ubuntu;ipv4 interface enp5s0: ubuntu
12/11/2024 09:14:32;0002;Comm@ubuntu;Svr;Comm@ubuntu;ipv6 interface lo: localhost
12/11/2024 09:14:32;0002;Comm@ubuntu;Svr;Comm@ubuntu;ipv6 interface enp5s0: ubuntu
12/11/2024 09:14:32;0002;Comm@ubuntu;Svr;Comm@ubuntu;/opt/pbs/sbin/pbs_comm ready (pid=2173384), Proxy Name:ubuntu:17001, Threads:4
12/11/2024 09:14:32;0000;Comm@ubuntu;Svr;Comm@ubuntu;Supported authentication method: resvport
12/11/2024 09:14:32;0c06;Comm@ubuntu;TPP;Comm@ubuntu(Thread 1);Thread ready
12/11/2024 09:14:32;0c06;Comm@ubuntu;TPP;Comm@ubuntu(Thread 0);Thread ready
12/11/2024 09:14:32;0c06;Comm@ubuntu;TPP;Comm@ubuntu(Thread 3);Thread ready
12/11/2024 09:14:32;0c06;Comm@ubuntu;TPP;Comm@ubuntu(Thread 2);Thread ready
12/11/2024 09:14:32;0c06;Comm@ubuntu;TPP;Comm@ubuntu(Thread 0);tpp_mbox_read;Unable to read from msg box
12/11/2024 09:14:32;0c06;Comm@ubuntu;TPP;Comm@ubuntu(Thread 0);tpp_mbox_read;Unable to read from msg box
12/11/2024 09:14:32;0c06;Comm@ubuntu;TPP;Comm@ubuntu(Thread 1);tpp_mbox_read;Unable to read from msg box
12/11/2024 09:14:32;0c06;Comm@ubuntu;TPP;Comm@ubuntu(Thread 1);tpp_mbox_read;Unable to read from msg box
12/11/2024 09:14:32;0c06;Comm@ubuntu;TPP;Comm@ubuntu(Thread 1);tpp_mbox_read;Unable to read from msg box
12/11/2024 09:14:32;0c06;Comm@ubuntu;TPP;Comm@ubuntu(Thread 1);tfd=14, Leaf registered address 172.16.4.1:15003
12/11/2024 09:14:34;0c06;Comm@ubuntu;TPP;Comm@ubuntu(Thread 1);tpp_mbox_read;Unable to read from msg box
12/11/2024 09:14:34;0c06;Comm@ubuntu;TPP;Comm@ubuntu(Thread 1);tpp_mbox_read;Unable to read from msg box
12/11/2024 09:14:34;0c06;Comm@ubuntu;TPP;Comm@ubuntu(Thread 1);tpp_mbox_read;Unable to read from msg box
12/11/2024 09:14:35;0c06;Comm@ubuntu;TPP;Comm@ubuntu(Thread 0);tpp_mbox_read;Unable to read from msg box
12/11/2024 09:14:35;0c06;Comm@ubuntu;TPP;Comm@ubuntu(Thread 0);tpp_mbox_read;Unable to read from msg box
12/11/2024 09:14:35;0c06;Comm@ubuntu;TPP;Comm@ubuntu(Thread 2);tpp_mbox_read;Unable to read from msg box
12/11/2024 09:14:35;0c06;Comm@ubuntu;TPP;Comm@ubuntu(Thread 2);tpp_mbox_read;Unable to read from msg box
12/11/2024 09:14:35;0c06;Comm@ubuntu;TPP;Comm@ubuntu(Thread 2);tpp_mbox_read;Unable to read from msg box
12/11/2024 09:14:35;0c06;Comm@ubuntu;TPP;Comm@ubuntu(Thread 2);tfd=16, Leaf registered address 172.16.4.1:15001
12/11/2024 09:14:36;0c06;Comm@ubuntu;TPP;Comm@ubuntu(Thread 1);tpp_mbox_read;Unable to read from msg box
12/11/2024 09:14:36;0c06;Comm@ubuntu;TPP;Comm@ubuntu(Thread 1);tpp_mbox_read;Unable to read from msg box
12/11/2024 09:14:36;0c06;Comm@ubuntu;TPP;Comm@ubuntu(Thread 1);tpp_mbox_read;Unable to read from msg box
12/11/2024 09:14:36;0c06;Comm@ubuntu;TPP;Comm@ubuntu(Thread 1);tpp_mbox_read;Unable to read from msg box
12/11/2024 09:14:36;0c06;Comm@ubuntu;TPP;Comm@ubuntu(Thread 2);tpp_mbox_read;Unable to read from msg box
12/11/2024 09:14:37;0c06;Comm@ubuntu;TPP;Comm@ubuntu(Thread 2);tpp_mbox_read;Unable to read from msg box
12/11/2024 09:14:37;0c06;Comm@ubuntu;TPP;Comm@ubuntu(Thread 2);tpp_mbox_read;Unable to read from msg box
12/11/2024 09:14:37;0c06;Comm@ubuntu;TPP;Comm@ubuntu(Thread 2);tfd=16, Connection from leaf 172.16.4.1:15001 down
12/11/2024 09:14:37;0c06;Comm@ubuntu;TPP;Comm@ubuntu(Thread 1);tpp_mbox_read;Unable to read from msg box
The last line is repeated every 10 seconds, so I didn’t post all those repeats. Similar errors show up in the /var/spool/pbs/mom_logs/20241211 file:
12/11/2024 09:21:18;0c06;pbs_mom;TPP;pbs_mom(Thread 0);tpp_mbox_read;Unable to read from msg box
12/11/2024 09:21:36;0002;pbs_mom;Svr;pbs_mom;HELLO sent to server at ubuntu:15001, stream:42
12/11/2024 09:21:36;0c06;pbs_mom;TPP;pbs_mom(Thread 0);tpp_mbox_read;Unable to read from msg box
12/11/2024 09:21:36;0c06;pbs_mom;TPP;pbs_mom(Thread 0);tpp_mbox_read;Unable to read from msg box
12/11/2024 09:21:36;0c06;pbs_mom;TPP;pbs_mom(Thread 0);tpp_mbox_read;Unable to read from msg box
12/11/2024 09:21:36;0001;pbs_mom;Svr;pbs_mom;im_eof, Premature end of message from addr 172.16.4.1:15001 on stream 42
12/11/2024 09:21:36;0002;pbs_mom;Svr;im_eof;Server closed connection.
I found this thread that says the error was addressed. I will reach out there too.