Can not start pbs_server (all other jobs are running)

Users/Site Administrators
1

I have installed pbs on a single node computer.

I run

sudo /etc/init.d/pbs start

Starting PBS
/opt/pbs/sbin/pbs_comm ready (pid=336099), Proxy Name:my-workstation:17001, Threads:4
PBS comm
PBS mom
PBS sched
Connecting to PBS dataservice...connected to PBS dataservice@my-workstation
PBS server

One thing I notice is that here it says my-workstation but it is actually called My-Workstation.

Then

sudo /etc/init.d/pbs start

pbs_server is not running
pbs_mom is pid 336109
pbs_sched is pid 336120
pbs_comm is 336099

The firewall is offline, I have added the static ip to /etc/hosts.

The content of /etc/pbs.conf is

PBS_SERVER=My-Workstation
PBS_START_SERVER=1
PBS_START_SCHED=1
PBS_START_COMM=1
PBS_START_MOM=1
PBS_EXEC=/opt/pbs
PBS_HOME=/var/spool/pbs
PBS_CORE_LIMIT=unlimited
PBS_SCP=/usr/bin/scp

How can I get the server running?

edit: Recent (clean) server log:

03/23/2022 14:32:16;0002;Server@my-workstation;Svr;Log;Log opened
03/23/2022 14:32:16;0002;Server@my-workstation;Svr;Server@my-workstation;pbs_version=20.0.0
03/23/2022 14:32:16;0002;Server@my-workstation;Svr;Server@my-workstation;pbs_build=mach=N/A:security=N/A:configure_args=N/A
03/23/2022 14:32:16;0002;Server@my-workstation;Svr;Server@my-workstation;hostname=my-workstation;pbs_leaf_name=N/A;pbs_mom_node_name=N/A
03/23/2022 14:32:16;0002;Server@my-workstation;Svr;Server@my-workstation;ipv4 interface lo: localhost
03/23/2022 14:32:16;0002;Server@my-workstation;Svr;Server@my-workstation;ipv4 interface enp42s0: My-Workstation
03/23/2022 14:32:16;0002;Server@my-workstation;Svr;Server@my-workstation;ipv6 interface lo: ip6-loopback
03/23/2022 14:32:16;0002;Server@my-workstation;Svr;Server@my-workstation;ipv6 interface enp42s0: My-Workstation
03/23/2022 14:32:16;0006;Server@my-workstation;Fil;Server@my-workstation;Version 20.0.0, started, initialization type = 1
03/23/2022 14:32:16;0002;Server@my-workstation;Svr;Server@my-workstation;pbs_status_db exit code 1
03/23/2022 14:32:16;0002;Server@my-workstation;Svr;Server@my-workstation;Starting PBS dataservice
03/23/2022 14:32:19;0002;Server@my-workstation;Svr;Server@my-workstation;connected to PBS dataservice@my-workstation
03/23/2022 14:32:19;0086;Server@my-workstation;Svr;pbs_python_ext_quick_start_interpreter;--> Python Interpreter quick started, compiled with version:'3.8.10 (default, Nov 26 2021, 20:14:08)
[GCC 9.3.0]' <--
03/23/2022 14:32:19;0086;Server@my-workstation;Svr;pbs_python_ext_quick_start_interpreter;--> Inserted Altair PBS Python modules dir '/opt/pbs/lib/python/altair' '/opt/pbs/lib/python/altair/pbs/v1'<--
03/23/2022 14:32:19;0086;Server@my-workstation;Svr;pbs_python_ext_quick_shutdown_interpreter;--> Stopping Python interpreter <--
03/23/2022 14:32:19;0d80;Server@my-workstation;TPP;Server@my-workstation(Main Thread);TPP authentication method = resvport
03/23/2022 14:32:19;0c06;Server@my-workstation;TPP;Server@my-workstation(Main Thread);TPP leaf node names = 130.83.24.92:15001,127.0.0.1:15001,130.83.24.92:15001
03/23/2022 14:32:19;0d80;Server@my-workstation;TPP;Server@my-workstation(Main Thread);Initializing TPP transport Layer
03/23/2022 14:32:19;0d80;Server@my-workstation;TPP;Server@my-workstation(Main Thread);Max files allowed = 16384
03/23/2022 14:32:19;0d80;Server@my-workstation;TPP;Server@my-workstation(Main Thread);TPP initialization done
03/23/2022 14:32:19;0d80;Server@my-workstation;TPP;Server@my-workstation(Main Thread);Connecting to pbs_comm My-Workstation:17001
03/23/2022 14:32:19;0c06;Server@my-workstation;TPP;Server@my-workstation(Thread 0);Thread ready
03/23/2022 14:32:19;0c06;Server@my-workstation;TPP;Server@my-workstation(Thread 0);Registering address 130.83.24.92:15001 to pbs_comm My-Workstation:17001
03/23/2022 14:32:19;0c06;Server@my-workstation;TPP;Server@my-workstation(Thread 0);Connected to pbs_comm My-Workstation:17001
03/23/2022 14:32:19;0002;Server@my-workstation;n/a;setup_env;read environment from /var/spool/pbs/pbs_environment
03/23/2022 14:32:19;0000;Server@my-workstation;Svr;Server@my-workstation;Supported authentication method: resvport
03/23/2022 14:32:19;0002;Server@my-workstation;Svr;Server@my-workstation;Stopping PBS dataservice
2

Please check and share the /var/spool/pbs/datastore/pg_log
after restarting pbs services one more time.

3

Thanks for your reply, the output is:

2022-03-27 09:53:04.292 CEST [473827] LOG: database system was shut down at 2022-03-24 12:05:41 CET
2022-03-27 09:53:04.296 CEST [473813] LOG: database system is ready to accept connections
2022-03-27 09:53:07.360 CEST [473813] LOG: received fast shutdown request
2022-03-27 09:53:07.366 CEST [473813] LOG: aborting any active transactions
2022-03-27 09:53:07.367 CEST [473813] LOG: background worker “logical replication launcher” (PID 473833) exited with exit code 1
2022-03-27 09:53:07.367 CEST [473828] LOG: shutting down
2022-03-27 09:53:07.390 CEST [473813] LOG: database system is shut down
2022-03-27 09:54:20.615 CEST [474266] LOG: database system was shut down at 2022-03-27 09:53:07 CEST
2022-03-27 09:54:20.620 CEST [474253] LOG: database system is ready to accept connections
2022-03-27 09:54:23.695 CEST [474253] LOG: received fast shutdown request
2022-03-27 09:54:23.704 CEST [474253] LOG: aborting any active transactions
2022-03-27 09:54:23.705 CEST [474253] LOG: background worker “logical replication launcher” (PID 474272) exited with exit code 1
2022-03-27 09:54:23.705 CEST [474267] LOG: shutting down
2022-03-27 09:54:23.725 CEST [474253] LOG: database system is shut down
2022-03-27 09:58:16.556 CEST [474666] LOG: database system was shut down at 2022-03-27 09:54:23 CEST
2022-03-27 09:58:16.561 CEST [474654] LOG: database system is ready to accept connections
2022-03-27 09:58:19.642 CEST [474654] LOG: received fast shutdown request
2022-03-27 09:58:19.651 CEST [474654] LOG: aborting any active transactions
2022-03-27 09:58:19.652 CEST [474654] LOG: background worker “logical replication launcher” (PID 474672) exited with exit code 1
2022-03-27 09:58:19.652 CEST [474667] LOG: shutting down
2022-03-27 09:58:19.672 CEST [474654] LOG: database system is shut down

4

Thank you @thephysicist for the logs, but they did not tell much about why it shut down.

Just to make sure:

  1. selinux is disabled and firewalls allow port 15001 to 15009 and 17001
  2. /etc/hosts have hostnames mapped to static IP address and DNS is resolvable
5

I do not think selinux is installed. The commands sestatus and getenforce are unknown, I do not have the /etc/selinux/config file (only /etc/selinux/semanage.conf and that does not contain anything useful as far as I can tell).
sudo ufw status
returns
Status: inactive.

/etc/hosts contains

127.0.0.1 localhost My-Workstation
130.83.24.92 My-Workstation

The latter ip is static and pingable.

6

Thank you @thephysicist .
I am not sure about this issue, might be other community members might chime in eventually.
Otherwise, If you can give a try to un-install and install it again, to see whether it made any difference.

7

@adarsh thank you for your time. I already tried reinstalling but that didn’t make any difference.

8

that’s a bit strange … can you verify that the ‘hostname’ command returns the same (“My-Workstation”)?

9

hostname returns it with correct capitalisation, i.e. My-Workstation.