Hostname issue setting up PBS Professional on CentOS 7

[CProvost]

Hello,

I’m trying to setup PBS Professional on the head node, running CentOS 7, of an OHPC cluster by following the steps outlined in this document:
https://github.com/openhpc/ohpc/releases/download/v1.3.8.GA/Install_guide-CentOS7-Warewulf-PBSPro-1.3.8-x86_64.pdf

Attempting to start the PBS service using:

systemctl start pbs

yields the status:

Other folks with a related problem usually seem to have some value following "Invalid local hostname: ", even if it’s incorrect, but my instance of the service doesn’t seem to be able to read anything.

Here are the contents of my /etc/pbs.conf file, /etc/hosts file, and the outputs from hostname and hostname -f:

Both SELinux and the firewall service are disabled:

I’d really appreciate any help, let me know if you need any more documentation.

[mkaro]

The value of PBS_LEAF_NAME should be the hostname of the interface over which you want PBS Pro to communicate, not the domain name. Unless you have multiple NICs, you should not need to set this value at all. Please try removing PBS_LEAF_NAME from /etc/pbs.conf and try starting PBS Pro.

[CProvost]

That worked perfectly, PBS started without issue following removal of that line.

Thanks!

[CProvost]

Hi again,

Upon restarting the service, I receive the following status:

I’m fairly sure this issue is unrelated to that originally presented in the thread, however the system setup remains same. Again, any help would be appreciated and let me know if you need more information.

Thanks

[mkaro]

Could you please try running the commands outside of systemctl? In other words (as root)…

# /opt/pbs/libexec/pbs_init.d stop
# /opt/pbs/libexec/pbs_init.d start

The output from these commands should help to diagnose the problem. It’s very unusual that the scheduler failed to start. After you stop PBS, please make sure there are no lingering processes (e.g. ps -ef | grep pbs).

[CProvost]

Stopping was successful and no related processes appeared to be lingering, (only grep result was the grep itself), however startup failed. Could this be an issue with permissions?

[mkaro]

Please run:

# ls -la /var/spool/pbs

and provide the output. PBS performs some sanity checks when it starts to make sure certain files and directories are present and have the expected permissions. Is /var/spool/pbs/sched_priv a symbolic link?

[CProvost]

/var/spool/pbs/sched_priv is not a symlink as can be seen from the ls, just a regular directory.

[mkaro]

It looks to be an issue with permissions, not on the directories themselves but how the filesystem is mounted. Please run:

# df -h /var/spool/pbs

That will tell you which volume the filesystem is mounted on. Then check the mount options for that filesystem using the “mount” command.

[CProvost]

Running:

# df -h /var/spool/pbs

yielded:

and the mount options for that filesystem are:

[mkaro]

I don’t see anything unusual in the output you provided. The one thing that puzzles me is the log message:

pbs_sched: Operation not permitted (1) in chk_file_sec, Security violation "/var/spool/pbs/sched_priv" resolves to "/var/spool/pbs"

I would have expected the two paths to be identical. I don’t believe xfs is a problem, though I tend to use ext4. Are there hidden attributes in XFS that could be affecting permissions?

You might try moving /var/spool/pbs (PBS_HOME in /etc/pbs.conf) to another filesystem and updating /etc/pbs.conf to point to the new location.

Ever seen this before @scc?

[CProvost]

Interestingly enough, a fresh attempt at initialization of the service appeared successful despite making no recent alterations. Restarting the service appeared equally successful. However, unless I am misunderstanding something, none of the functions of the service appear operational:

User liuxiapiao appeared to be receiving a message similar to “(to postgres) root on none” and resolved their issue by disabling their firewall.

http://community.openpbs.org/t/pbs-execution-host-couldnt-talk-to-pbs-server/1280

Our firewall is definitely disabled, but could this point to some other disruption of communication between the command line and the pbs server? Also, could this in any way be related to the previous issue with the scheduler?

I’ve checked and the server is active:

[mkaro]

What happens when you telnet to the server port?

$ telnet -4 pbs-server 15001
Trying 192.168.1.2...
Connected to pbs-server.
Escape character is '^]'.

+2+15+15056+0+1Connection closed by foreign host.

When I try connecting to an unused port I see this:

$ telnet -4 pbs-server 15005
Trying 192.168.1.2...
telnet: connect to address 192.168.1.2: Connection refused

[jiaqi]

Hi, everyone.
I am install pbs pro in ubuntu 18.04 in a single node.
I have come across this error when I tried to start the pbs server.
$ sudo /etc/init.d/pbs start
Starting PBS
/opt/pbs/sbin/pbs_comm ready (pid=26546), Proxy Name:localhost:17001, Threads:4
PBS comm
PBS mom
PBS sched
/opt/pbs/sbin/pbs_ds_systemd: 43: [: xdegraded: unexpected operator
Connecting to PBS dataservice…connected to PBS dataservice@localhost
PBS server

#And When I type
$ /etc/init.d/pbs status
pbs_server is not running
pbs_mom is pid 26575
pbs_sched is not running
pbs_comm is not running

Could you suggest what might be the problem?
Thank you.

Wishes,
Jiaqi

[subhasisb]

Hi,

Welcome to the openpbs community!

The first step would be to look at the server logs. They are located at /var/spool/pbs/server_logs/… by default