# Failover time very long and job IDs lost **URL:** \ **Category:** Users/Site Administrators\ **Created:** [September 20, 2023, 6:01am UTC](https://community.openpbs.org/t/failover-time-very-long-and-job-ids-lost/3670 "2023-09-20T06:01:45Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![tuthmose](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/tuthmose/32/635_2.png) [@tuthmose](https://community.openpbs.org/u/tuthmose)\ **Post date:** [September 20, 2023, 6:01am UTC](https://community.openpbs.org/t/failover-time-very-long-and-job-ids-lost/3670/1 "2023-09-20T06:01:45Z")
Hi all, a few days ago we a PBS 22.05 installation on Rocky 8.8 switched from primary to secondary server because primary VM was having problem. The mechanism worked but it took a very long time to finish, about 30 minutes. When I manually tested it a few months ago (with about half of the job present in logs or db) it took the 7-10 minutes that are also given as a rule of thumb on the docs. The spool area is a shared NFS mounted with `rw,sync,soft` options. What can be the source of the slowdown? Should I expect it to worsen over time? After we solved problems the primary wen active once again and we still had to wait about half an hour. Also, during the fail over, some jobs changed prefix i. e. before the fail over a job could be named ```auto 12345.primary.mydns ``` while after it is ```auto 56789.secondary.mydns ``` if I now try to inquire about the second job with `qstat` I got “Unknown job”. It is present in the logs. What happened? Thank you for any support ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [September 24, 2023, 8:20pm UTC](https://community.openpbs.org/t/failover-time-very-long-and-job-ids-lost/3670/2 "2023-09-24T20:20:34Z")
Might be some of these factors 1. Job history 2. Jobs in the queue 3. Performance of the disk hosting $PBS\_HOME ( SSD or networked drive or SAN ) 4. Please note the job id will always be .PBS\_PRIMARY ( even when the secondary is active) There would be some interval, where the client commands would not be rendered. However, once the services are recovered and nodes know that now secondary is the active server, there will be some client command query disruption. ---
**Author:** ![tuthmose](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/tuthmose/32/635_2.png) [@tuthmose](https://community.openpbs.org/u/tuthmose)\ **Post date:** [September 25, 2023, 7:21am UTC](https://community.openpbs.org/t/failover-time-very-long-and-job-ids-lost/3670/3 "2023-09-25T07:21:41Z")
Hi Adarsh, thank you for your reply. About point 1 do you suggest to disable job history or reduce log verbosity? If this help it would point to problems with point 3. About point 2 there were about 200 hundred jobs; I’d guess it’s not a big load; primary and secondary are VMs with 4 cores and 2 GB of RAM. About query disruption, what do you mean? ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [September 25, 2023, 8:39am UTC](https://community.openpbs.org/t/failover-time-very-long-and-job-ids-lost/3670/4 "2023-09-25T08:39:41Z")
Please check this guide **G-370 and 371 PBS Professional 2022.1 Administrator’s Guide** ![image](https://canada1.discourse-cdn.com/flex030/uploads/pbspro/original/1X/3867b99b9a83ced7c5bed9b274cec1543566235e.png) and ![image](https://canada1.discourse-cdn.com/flex030/uploads/pbspro/original/1X/1c41c8e15c7057c7c114ff101afac6cd2ba93a9b.png) ---
**Author:** ![tuthmose](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/tuthmose/32/635_2.png) [@tuthmose](https://community.openpbs.org/u/tuthmose)\ **Post date:** [September 27, 2023, 5:58am UTC](https://community.openpbs.org/t/failover-time-very-long-and-job-ids-lost/3670/5 "2023-09-27T05:58:00Z")
Hi Adarsh, thank you for the note about `/tmp/.pbsrc.UID` which I did not know or remember. As for the failover time which the AG too estimates to a few minutes there is evidently something awry with our config. I’ll keep you posted when/if there’s a chance to investigate. ---
**Author:** ![tuthmose](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/tuthmose/32/635_2.png) [@tuthmose](https://community.openpbs.org/u/tuthmose)\ **Post date:** [November 7, 2023, 3:23pm UTC](https://community.openpbs.org/t/failover-time-very-long-and-job-ids-lost/3670/6 "2023-11-07T15:23:49Z")
Hi, to provide a follow up: it was a trivial problem: the NFS area was mounted with `sync`; my fault (I even wrote it in the first message). Now the failover is very fast, within a few minutes. Thank you