# Failover Setup Issues
**URL:** https://community.openpbs.org/t/failover-setup-issues/1534
**Category:** Users/Site Administrators
**Created:** [March 29, 2019, 7:59pm UTC](https://community.openpbs.org/t/failover-setup-issues/1534 "2019-03-29T19:59:59Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@rgarg](https://community.openpbs.org/u/rgarg)
#### Post date: [March 29, 2019, 7:59pm UTC](https://community.openpbs.org/t/failover-setup-issues/1534/1 "2019-03-29T19:59:59Z")
Hi,
Below are the details on how I had worked on Failover Setup.
In Open Stack I had created four instances
Instance1 - Primary Server(PS)
Instance2 - Secondary Server(SS)
Instance3 - Node1(i.e execution host)
Instance4 - NFS Server
Case1: When NFS is not being used.
I had configured Primary Server, Secondary Server and Node1 as per the details mentioned in PBS Admin Guide Section 9 for failover setup. PS, SS and Node1 are able to see each other.
At first I had submitted some jobs on the PS and they were executed successfully. Secondly I had stopped PBS on PS within few seconds I could notice that SS has taken over the control and its is unable to execute the jobs. This signifies we need NFS
Case2: When NFS is being used
-Created a PBS\_HOME directory which is now hard mounted on PS, SS, Node1 and NFS Server.
-Stopped PBS on PS, SS and Node1
-Did path changes in /etc/pbs.conf for pointing PBS\_HOME variable to Shared File System path
-Copied all the files from /var/spool/pbs to /PBS\_HOME/
-Configurations for /etc/hosts is also in sync
## When I start PBS its unable to start neither on PS, SS and Node1 Below is the error message: Mar 29 19:30:55 primaryserver.novalocal pbs\_init.d[17147]: pbs\_sched startup failed, exit 1 aborting. Mar 29 19:30:55 primaryserver.novalocal systemd[1]: pbs.service: control process exited, code=exited status=1 Mar 29 19:30:55 primaryserver.novalocal systemd[1]: Failed to start Portable Batch System. – Subject: Unit pbs.service has failed – Defined-By: systemd – Support: [http://lists.freedesktop.org/mailman/listinfo/systemd-devel](http://lists.freedesktop.org/mailman/listinfo/systemd-devel)
## – Unit pbs.service has failed.
– The result is failed.
Mar 29 19:30:55 primaryserver.novalocal systemd[1]: Unit pbs.service entered failed state.
Mar 29 19:30:55 primaryserver.novalocal systemd[1]: pbs.service failed.
Mar 29 19:30:55 primaryserver.novalocal polkitd[3253]: Unregistered Authentication Agent for unix-process:17141:1425045 (system bus name :1.132, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en\_US.UTF-8) (disconnect)
While troubleshooting I had encountered below type of message as well
/etc/init.d/pbs start
Starting PBS
PBS comm
/opt/pbs/sbin/pbs\_comm ready (pid=12782), Proxy Name:secondaryserver:17001, Threads:4
pbs\_sched: Permission denied (13) in chk\_file\_sec, Security violation “/PBS\_HOME/pbs/sched\_priv” resolves to “/PBS\_HOME”
pbs\_sched startup failed, exit 1 aborting.
I had made sure that UID is same for PS, SS, Node1 and NFS Server and also shared file system directory has root privileges.
Any leads will be helpful
---
### Author:  [@adarsh](https://community.openpbs.org/u/adarsh)
#### Post date: [March 30, 2019, 9:08pm UTC](https://community.openpbs.org/t/failover-setup-issues/1534/2 "2019-03-30T21:08:35Z")
For Failover:
Yes shared file system with file locking mechanism is required for PBS Pro failover setup.
**Important:** Make sure NFS lock services are up and running
Please follow these steps:
**Primary Server:**
- install PBS Server with PBS\_HOME directory on the NFS
- submit couple of jobs, to make sure setup works by submitting couple of jobs
- stop the PBS Services
**Secondary Server:**
- install PBS server with PBS\_HOME on the local disk
- stop the PBS Services
- edit the /etc/pbs.conf and point it to PBS\_HOME setup on the NFS
- start the PBS Services
- make sure setup works by submitting couple of jobs
- stop the PBS Services
**Note: When starting the services**
- start the primary pbs server first
- start the second pbs server next
**When stopping the services**
- stop the secondary pbs server first
- stop the primary pbs server next
Caution: Never try to stop and start PBS Services on the primary and secondary in short span of time, always make sure there is some time gap between starting and stopping (vice versa) the services.
It seems the file/folder permissions of the PBS\_HOME and sub directories are disturbed. Please follow the above steps, it would work without any issues.
---
### Author:  [@rgarg](https://community.openpbs.org/u/rgarg)
#### Post date: [March 31, 2019, 1:14am UTC](https://community.openpbs.org/t/failover-setup-issues/1534/3 "2019-03-31T01:14:15Z")
Hi Adarsh,
Thanks for the details. I will do the setup from scratch and will let you know in case I encounter any issues.
Thanks,
Rakhen
---
### Author:  [@rgarg](https://community.openpbs.org/u/rgarg)
#### Post date: [April 1, 2019, 7:00pm UTC](https://community.openpbs.org/t/failover-setup-issues/1534/4 "2019-04-01T19:00:06Z")
Hi Adarsh,
Below are the steps that I had followed for configuring failover:
1.Created NFS Server with the shared directory named as PBS\_HOME
2.Hard mounted shared NFS folder on primary server instance
3.Installed PBS Pro server package by pointing to shared folder.
When I start PBS Server I am facing the below issue
---
### Author:  [@adarsh](https://community.openpbs.org/u/adarsh)
#### Post date: [April 2, 2019, 7:44am UTC](https://community.openpbs.org/t/failover-setup-issues/1534/5 "2019-04-02T07:44:41Z")
Rakhen,
Before the installation did you update the /etc/pbs.conf for PBS\_HOME to point to the NFS location .
Are you sure you removed the remnants of the previous installation ?
Note: The location of PBS\_HOME is specified in the file /etc/pbs.conf, but defaults to /var/spool/pbs if not specified. The default for PBS\_EXEC is /opt/pbs. You can specify a non-default location for PBS\_EXEC via the --prefix option to rpm when installing the new PBS.
Thank you
---
### Author:  [@rgarg](https://community.openpbs.org/u/rgarg)
#### Post date: [April 3, 2019, 1:38pm UTC](https://community.openpbs.org/t/failover-setup-issues/1534/6 "2019-04-03T13:38:00Z")
Hi Adarsh,
I did a fresh start of the entire setup. Actually I had exported the value for PBS\_HOME so that it points to the NFS location. As I could see before installing the server package there isn’t any /etc/pbs.conf present.
Thanks
---
### Author:  [@rgarg](https://community.openpbs.org/u/rgarg)
#### Post date: [April 3, 2019, 5:35pm UTC](https://community.openpbs.org/t/failover-setup-issues/1534/7 "2019-04-03T17:35:28Z")
Hi Adarsh,
NFS Server Setup:
vi /etc/selinux/config (Changed to disabled)
yum -y install nfs-utils libnfsidmap
systemctl enable rpcbind
systemctl start nfs-server
systemctl start rpc-statd
systemctl start nfs-idmapd
systemctl start nfslock
mkdir /PBS\_HOME
chmod 777 /PBS\_HOME/
vi /etc/exports Adding the text in () to /etc/exports (/PBS\_HOME \*(rw,sync,no\_root\_squash))
exportfs -r
showmount -e localhost
systemctl restart nfs-server
PrimaryServer Setup
1. Setting up NFS client on Primary Server
vi /etc/selinux/config (Make sure selinux is disabled)
yum -y install nfs-utils libnfsidmap
systemctl enable rpcbind
systemctl start rpcbind
systemctl enable nfslock
systemctl enable nfs-server
systemctl start nfs-server
systemctl enable nfs-idmap
systemctl start nfs-idmap
systemctl start nfslock
mkdir /PBS\_HOME
mount -o rw,hard,intr :/PBS\_HOME /PBS\_HOME ----For Hard mounting
df -kh
cd /PBS\_HOME/
ll
touch test
ll
vi /etc/fstab Adding the text in () in /etc/fstab (:/PBS\_HOME /PBS\_HOME nfs rw,sync,hard,intr 0 0)
cd
umount /PBS\_HOME/
mount -av
df -kh
reboot
2. Installing Server Package
useradd --system -m pbsdata
wget [https://github.com/PBSPro/pbspro/releases/download/v18.1.4/pbspro\_1.8.4.centos7.zip](https://github.com/PBSPro/pbspro/releases/download/v18.1.4/pbspro_1.8.4.centos7.zip)
unzip pbspro\_1.8.4.centos7.zip
cp pbspro\_1.8.4.centos7/pbspro-server-18.1.4-0.x86\_64.rpm ~
export PBS\_DATA\_SERVICE\_USER=pbsdata
export PBS\_HOME=/PBS\_HOME
yum install pbspro-server-18.1.4-0.x86\_64.rpm
vi /etc/pbs.conf (Check whether parameters are in thsi way
PBS\_EXEC=/opt/pbs
PBS\_HOME=/PBS\_HOME
PBS\_SERVER=primaryserver
PBS\_START\_SERVER=1
PBS\_START\_SCHED=1
PBS\_START\_COMM=1
PBS\_START\_MOM=0
PBS\_CORE\_LIMIT=unlimited
PBS\_SCP=/bin/scp
)
vi /PBS\_HOME/mom\_priv/config (CHANGE\_THIS\_TO\_PBS\_PRO\_SERVER\_HOSTNAME to your respective hostname)
vi /etc/hosts (Add your hostname)
/etc/init.d/pbs start
[root@primaryserver ~]# /etc/init.d/pbs start
Starting PBS
PBS Home directory /PBS\_HOME needs updating.
Running /opt/pbs/libexec/pbs\_habitat to update it.
* * *
\*\*\* Error initializing the PBS dataservice
Error details:
Creating the PBS Data Service…
Starting PBS Data Service…
pg\_ctl: could not start server
Examine the log output.
Failed to start PBS Data Service
Error starting PBS Data Service
rm: cannot remove ‘/PBS\_HOME/datastore’: Directory not empty
[root@primaryserver ~]# cd /PBS\_HOME/
[root@primaryserver PBS\_HOME]# ls -l
total 4
drwxr-xr-x 2 root root 6 Apr 3 14:16 aux
drwx------ 2 root root 6 Apr 3 14:16 checkpoint
drwxr-xr-x 2 root root 6 Apr 3 14:16 comm\_logs
drwx------ 2 pbsdata root 42 Apr 3 17:22 datastore
drwxr-xr-x 2 root root 6 Apr 3 14:16 mom\_logs
drwxr-x–x 4 root root 45 Apr 3 14:19 mom\_priv
-rw-r–r-- 1 root root 19 Apr 3 14:16 pbs\_environment
drwxr-xr-x 2 root root 6 Apr 3 14:16 sched\_logs
drwxr-x— 2 root root 86 Apr 3 14:16 sched\_priv
drwxr-xr-x 2 root root 6 Apr 3 14:16 server\_logs
drwxr-x— 6 root root 77 Apr 3 14:16 server\_priv
drwxrwxrwt 2 root root 6 Apr 3 17:22 spool
drwxrwxrwt 2 root root 6 Apr 3 14:16 undelivered
I have even tried chown postgres:root datastore. When I start PBS again its reverted back to pbsdata and fails as well. Also removed datastore and started again still same issue persists.
Any idea where I am wrong?
Thanks
---
### Author:  [@adarsh](https://community.openpbs.org/u/adarsh)
#### Post date: [April 5, 2019, 4:47pm UTC](https://community.openpbs.org/t/failover-setup-issues/1534/8 "2019-04-05T16:47:21Z")
Tested your configuration a minute ago and it worked for me:
```
[root@headnode ~]# cat /etc/exports
/home *(rw,sync,no_root_squash,no_subtree_check)
/app *(rw,sync,no_root_squash,subtree_check)
[root@headnode ~]# cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)
[root@headnode ~]#ls -ltr /
drwxr-xr-x 17 root root 295 Apr 5 17:39 app
[root@n1 pbspro_1.8.4.centos7]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Tue Aug 14 10:37:36 2018
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/centos-root / xfs defaults 0 0
UUID=5986bc7d-5e77-4e9a-90b3-257778d6318f /boot xfs defaults 0 0
/dev/mapper/centos-swap swap swap defaults 0 0
headnode:/app /app nfs defaults 0 0
[root@n1 pbspro_1.8.4.centos7]# ls -ltr /
drwxr-xr-x 17 root root 295 Apr 5 17:39 app
[root@n1 ~]# ls
anaconda-ks.cfg Desktop Documents Downloads initial-setup-ks.cfg Music pbspro_1.8.4.centos7 pbspro_1.8.4.centos7.zip Pictures Public Templates Videos
[root@n1 ~]# cd pbspro_1.8.4.centos7/
[root@n1 pbspro_1.8.4.centos7]# ls
COPYRIGHT LICENSE pbspro-client-18.1.4-0.x86_64.rpm pbspro-debuginfo-18.1.4-0.x86_64.rpm pbspro-execution-18.1.4-0.x86_64.rpm pbspro-server-18.1.4-0.x86_64.rpm README.md
[root@n1 pbspro_1.8.4.centos7]# cat /etc/pbs.conf
PBS_EXEC=/opt/pbs
PBS_HOME=/app
PBS_SERVER=n1
PBS_START_SERVER=1
PBS_START_SCHED=1
PBS_START_COMM=1
PBS_START_MOM=0
PBS_CORE_LIMIT=unlimited
PBS_SCP=/usr/bin/scp
[root@n1 pbspro_1.8.4.centos7]# yum install pbspro-server-18.1.4-0.x86_64.rpm
Loaded plugins: fastestmirror, langpacks
Examining pbspro-server-18.1.4-0.x86_64.rpm: pbspro-server-18.1.4-0.x86_64
Marking pbspro-server-18.1.4-0.x86_64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package pbspro-server.x86_64 0:18.1.4-0 will be installed
--> Processing Dependency: perl(Env) for package: pbspro-server-18.1.4-0.x86_64
Loading mirror speeds from cached hostfile
* base: mirror.mhd.uk.as44574.net
* epel: epel.mirror.wearetriple.com
* extras: mirror.econdc.com
* updates: mirror.econdc.com
--> Processing Dependency: perl(Switch) for package: pbspro-server-18.1.4-0.x86_64
--> Processing Dependency: postgresql-server for package: pbspro-server-18.1.4-0.x86_64
--> Processing Dependency: tcl for package: pbspro-server-18.1.4-0.x86_64
--> Processing Dependency: tk for package: pbspro-server-18.1.4-0.x86_64
--> Processing Dependency: libhwloc.so.5()(64bit) for package: pbspro-server-18.1.4-0.x86_64
--> Processing Dependency: libpq.so.5()(64bit) for package: pbspro-server-18.1.4-0.x86_64
--> Processing Dependency: libtcl8.5.so()(64bit) for package: pbspro-server-18.1.4-0.x86_64
--> Processing Dependency: libtk8.5.so()(64bit) for package: pbspro-server-18.1.4-0.x86_64
--> Running transaction check
---> Package hwloc-libs.x86_64 0:1.11.8-4.el7 will be installed
---> Package perl-Env.noarch 0:1.04-2.el7 will be installed
---> Package perl-Switch.noarch 0:2.16-7.el7 will be installed
---> Package postgresql-libs.x86_64 0:9.2.24-1.el7_5 will be installed
---> Package postgresql-server.x86_64 0:9.2.24-1.el7_5 will be installed
--> Processing Dependency: postgresql(x86-64) = 9.2.24-1.el7_5 for package: postgresql-server-9.2.24-1.el7_5.x86_64
---> Package tcl.x86_64 1:8.5.13-8.el7 will be installed
---> Package tk.x86_64 1:8.5.13-6.el7 will be installed
--> Running transaction check
---> Package postgresql.x86_64 0:9.2.24-1.el7_5 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
=====================================================================================================================================================================================
Package Arch Version Repository Size
=====================================================================================================================================================================================
Installing:
pbspro-server x86_64 18.1.4-0 /pbspro-server-18.1.4-0.x86_64 18 M
Installing for dependencies:
hwloc-libs x86_64 1.11.8-4.el7 base 1.6 M
perl-Env noarch 1.04-2.el7 base 16 k
perl-Switch noarch 2.16-7.el7 base 22 k
postgresql x86_64 9.2.24-1.el7_5 base 3.0 M
postgresql-libs x86_64 9.2.24-1.el7_5 base 234 k
postgresql-server x86_64 9.2.24-1.el7_5 base 3.8 M
tcl x86_64 1:8.5.13-8.el7 base 1.9 M
tk x86_64 1:8.5.13-6.el7 base 1.4 M
Transaction Summary
=====================================================================================================================================================================================
Install 1 Package (+8 Dependent packages)
Total size: 30 M
Total download size: 12 M
Installed size: 61 M
Is this ok [y/d/N]: y
Downloading packages:
(1/8): perl-Env-1.04-2.el7.noarch.rpm | 16 kB 00:00:00
(2/8): perl-Switch-2.16-7.el7.noarch.rpm | 22 kB 00:00:00
(3/8): postgresql-libs-9.2.24-1.el7_5.x86_64.rpm | 234 kB 00:00:00
(4/8): postgresql-9.2.24-1.el7_5.x86_64.rpm | 3.0 MB 00:00:01
(5/8): tcl-8.5.13-8.el7.x86_64.rpm | 1.9 MB 00:00:01
(6/8): hwloc-libs-1.11.8-4.el7.x86_64.rpm | 1.6 MB 00:00:01
(7/8): tk-8.5.13-6.el7.x86_64.rpm | 1.4 MB 00:00:01
(8/8): postgresql-server-9.2.24-1.el7_5.x86_64.rpm | 3.8 MB 00:00:01
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 6.0 MB/s | 12 MB 00:00:02
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : postgresql-libs-9.2.24-1.el7_5.x86_64 1/9
Installing : 1:tcl-8.5.13-8.el7.x86_64 2/9
Installing : 1:tk-8.5.13-6.el7.x86_64 3/9
Installing : postgresql-9.2.24-1.el7_5.x86_64 4/9
Installing : postgresql-server-9.2.24-1.el7_5.x86_64 5/9
Installing : hwloc-libs-1.11.8-4.el7.x86_64 6/9
Installing : perl-Switch-2.16-7.el7.noarch 7/9
Installing : perl-Env-1.04-2.el7.noarch 8/9
Installing : pbspro-server-18.1.4-0.x86_64 9/9
*** PBS Installation Summary
***
*** Postinstall script called as follows:
*** /opt/pbs/libexec/pbs_postinstall server 18.1.4 /opt/pbs /var/spool/pbs postgres
***
*** Existing configuration file found: /etc/pbs.conf
***
*** Saving /etc/pbs.conf as /etc/pbs.conf.pre.18.1.4
*** Replacing /etc/pbs.conf with /etc/pbs.conf.18.1.4
*** /etc/pbs.conf has been modified.
*** The original contents have been saved to /etc/pbs.conf.pre.18.1.4
***
*** Registering PBS Pro as a service.
Created symlink from /etc/systemd/system/multi-user.target.wants/pbs.service to /usr/lib/systemd/system/pbs.service.
***
*** PBS_HOME is /app
*** Creating new file /app/pbs_environment
*** WARNING: TZ not set in /app/pbs_environment
***
*** The PBS Pro server has been installed in /opt/pbs/sbin.
*** The PBS Pro scheduler has been installed in /opt/pbs/sbin.
***
*** The PBS Pro communication agent has been installed in /opt/pbs/sbin.
***
*** The PBS Pro MOM has been installed in /opt/pbs/sbin.
***
*** The PBS commands have been installed in /opt/pbs/bin.
***
*** End of /opt/pbs/libexec/pbs_postinstall
Verifying : perl-Env-1.04-2.el7.noarch 1/9
Verifying : 1:tcl-8.5.13-8.el7.x86_64 2/9
Verifying : perl-Switch-2.16-7.el7.noarch 3/9
Verifying : hwloc-libs-1.11.8-4.el7.x86_64 4/9
Verifying : 1:tk-8.5.13-6.el7.x86_64 5/9
Verifying : postgresql-server-9.2.24-1.el7_5.x86_64 6/9
Verifying : postgresql-libs-9.2.24-1.el7_5.x86_64 7/9
Verifying : postgresql-9.2.24-1.el7_5.x86_64 8/9
Verifying : pbspro-server-18.1.4-0.x86_64 9/9
Installed:
pbspro-server.x86_64 0:18.1.4-0
Dependency Installed:
hwloc-libs.x86_64 0:1.11.8-4.el7 perl-Env.noarch 0:1.04-2.el7 perl-Switch.noarch 0:2.16-7.el7 postgresql.x86_64 0:9.2.24-1.el7_5 postgresql-libs.x86_64 0:9.2.24-1.el7_5
postgresql-server.x86_64 0:9.2.24-1.el7_5 tcl.x86_64 1:8.5.13-8.el7 tk.x86_64 1:8.5.13-6.el7
Complete!
[root@n1 pbspro_1.8.4.centos7]# /etc/init.d/pbs start
Starting PBS
PBS Home directory /app needs updating.
Running /opt/pbs/libexec/pbs_habitat to update it.
***
*** Setting default queue and resource limits.
***
Connecting to PBS dataservice....connected to PBS dataservice@n1
*** End of /opt/pbs/libexec/pbs_habitat
Home directory /app updated.
/opt/pbs/sbin/pbs_comm ready (pid=40824), Proxy Name:n1.lab.com:17001, Threads:4
PBS comm
Creating usage database for fairshare.
PBS sched
Connecting to PBS dataservice.....connected to PBS dataservice@n1
Licenses valid for 10000000 Floating hosts
PBS server
[root@n1 pbspro_1.8.4.centos7]# qstat -Bf
Server: n1
server_state = Active
server_host = n1.lab.com
scheduling = True
total_jobs = 0
state_count = Transit:0 Queued:0 Held:0 Waiting:0 Running:0 Exiting:0 Begun
:0
default_queue = workq
log_events = 511
mail_from = adm
query_other_jobs = True
resources_default.ncpus = 1
default_chunk.ncpus = 1
scheduler_iteration = 600
FLicenses = 20000000
resv_enable = True
node_fail_requeue = 310
max_array_size = 10000
pbs_license_min = 0
pbs_license_max = 2147483647
pbs_license_linger_time = 31536000
license_count = Avail_Global:10000000 Avail_Local:10000000 Used:0 High_Use:
0
pbs_version = 18.1.4
eligible_time_enable = False
max_concurrent_provision = 5
power_provisioning = False
```
---
### Author:  [@rgarg](https://community.openpbs.org/u/rgarg)
#### Post date: [April 5, 2019, 5:02pm UTC](https://community.openpbs.org/t/failover-setup-issues/1534/9 "2019-04-05T17:02:39Z")
Apart from /app \*(rw,sync,no\_root\_squash,subtree\_check) have you made any changes to the steps that I had listed?
---
### Author:  [@adarsh](https://community.openpbs.org/u/adarsh)
#### Post date: [April 5, 2019, 6:56pm UTC](https://community.openpbs.org/t/failover-setup-issues/1534/10 "2019-04-05T18:56:13Z")
Rakhen,
No changes were made to any folders or directories.
The above command line protocol were executed as they were.
/app was already available on n1 ( i did not run the exportfs , mount )
Alternatively, could you please try to install locally on the system & start the services & stop the services, then copy $PBS\_HOME (do not move) by preserving permissions to /PBS\_HOME , update PBS\_HOME in /etc/pbs.conf and start the services.
Thank you
---
### Author:  [@rgarg](https://community.openpbs.org/u/rgarg)
#### Post date: [April 10, 2019, 1:30pm UTC](https://community.openpbs.org/t/failover-setup-issues/1534/11 "2019-04-10T13:30:20Z")
Hi Adarsh,
I had successfully configured Failover Setup. Just added subtree\_check additionally to the steps listed above.
Thanks for your support…