# PP-832: Which scheduler to talk to while taking over from Primary **URL:** \ **Category:** Developers\ **Created:** [March 12, 2018, 10:06am UTC](https://community.openpbs.org/t/pp-832-which-scheduler-to-talk-to-while-taking-over-from-primary/889 "2018-03-12T10:06:25Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![prakashcv13](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/prakashcv13/32/65_2.png) [@prakashcv13](https://community.openpbs.org/u/prakashcv13)\ **Post date:** [March 12, 2018, 10:06am UTC](https://community.openpbs.org/t/pp-832-which-scheduler-to-talk-to-while-taking-over-from-primary/889/1 "2018-03-12T10:06:25Z")
Hi All, [PP-832](https://pbspro.atlassian.net/browse/PP-832) presents an issue in a failover setup whereby the secondary while taking over from the primary checks **only once** if it can communicate with the scheduler on the primary host or not and proceeds accordingly. I have written a summary of this issue and proposed two approaches to fix it over [here](https://pbspro.atlassian.net/wiki/spaces/PD/pages/261193738/PP-832+PBSPro+failover+secondary+server+fails+to+continuously+check+whether+it+needs+to+start+a+scheduler+locally). I request the community to provide feedback. Thanks, Prakash ---
**Author:** ![suresht](https://avatars.discourse-cdn.com/v4/letter/s/ec9cab/32.png) [@suresht](https://community.openpbs.org/u/suresht)\ **Post date:** [March 14, 2018, 10:11am UTC](https://community.openpbs.org/t/pp-832-which-scheduler-to-talk-to-while-taking-over-from-primary/889/2 "2018-03-14T10:11:49Z")
@prakashcv13, Thanks for writing the EDD and I vote for solution 1. Second step in in this solution is - Any time the scheduler on the primary goes down, the secondary server will spawn a local scheduler. I am assuming primary server is up in this case even though the scheduler went down. If yes just wanted to know why can’t the primary server itself start the scheduler locally (This way we don’t lose fairshare usage for one cycle, other configuration of sched etc) If the second step in solution-1 applies only when secondary becomes active then we need to rephrase the above statement so as to reflect this. Because Multisched is already checked in and If we are not considering Multisched we need to mention in EDD that it applies only to default scheduler for now but it can be enhanced to consider multiple schedulers as part of Multisched failover interface. ---
**Author:** ![prakashcv13](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/prakashcv13/32/65_2.png) [@prakashcv13](https://community.openpbs.org/u/prakashcv13)\ **Post date:** [March 27, 2018, 6:37am UTC](https://community.openpbs.org/t/pp-832-which-scheduler-to-talk-to-while-taking-over-from-primary/889/3 "2018-03-27T06:37:53Z")
> [@suresht](#): > > I am assuming primary server is up in this case even though the scheduler went down. If yes just wanted to know why can’t the primary server itself start the scheduler locally (This way we don’t lose fairshare usage for one cycle, other configuration of sched etc) > > If the second step in solution-1 applies only when secondary becomes active then we need to rephrase the above statement so as to reflect this. I see that I had not phrased step 2 in solution-1 properly, have corrected it now. > [@suresht](#): > > Because Multisched is already checked in and If we are not considering Multisched we need to mention in EDD that it applies only to default scheduler for now but it can be enhanced to consider multiple schedulers as part of Multisched failover interface. done! Thanks, Prakash ---
**Author:** ![agrawalravi90](https://avatars.discourse-cdn.com/v4/letter/a/848f3c/32.png) [@agrawalravi90](https://community.openpbs.org/u/agrawalravi90)\ **Post date:** [April 2, 2018, 7:36am UTC](https://community.openpbs.org/t/pp-832-which-scheduler-to-talk-to-while-taking-over-from-primary/889/4 "2018-04-02T07:36:52Z")
Thanks for starting this discussion Prakash. I like solution 2 better, the secondary should just spawn its own scheduler, it’s cleaner and more robust than solution 1 in my opinion. ---
**Author:** ![Klovely](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/klovely/32/117_2.png) [@Klovely](https://community.openpbs.org/u/Klovely)\ **Post date:** [April 2, 2018, 12:10pm UTC](https://community.openpbs.org/t/pp-832-which-scheduler-to-talk-to-while-taking-over-from-primary/889/5 "2018-04-02T12:10:21Z")
Hi Prakash , Thanks for this discussion , I vote for solution 1 because with solution 1 we keep communicating with the scheduler on the primary host and that will preserve run-time fairshare information. Regards, Lovely ---
**Author:** ![prakashcv13](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/prakashcv13/32/65_2.png) [@prakashcv13](https://community.openpbs.org/u/prakashcv13)\ **Post date:** [April 3, 2018, 5:18am UTC](https://community.openpbs.org/t/pp-832-which-scheduler-to-talk-to-while-taking-over-from-primary/889/6 "2018-04-03T05:18:06Z")
Thank you @Klovely and @agrawalravi90 for the inputs. fairshare is the main reason that the secondary tries connecting to the scheduler on the primary host. ---
**Author:** ![agrawalravi90](https://avatars.discourse-cdn.com/v4/letter/a/848f3c/32.png) [@agrawalravi90](https://community.openpbs.org/u/agrawalravi90)\ **Post date:** [April 4, 2018, 12:56am UTC](https://community.openpbs.org/t/pp-832-which-scheduler-to-talk-to-while-taking-over-from-primary/889/7 "2018-04-04T00:56:34Z")
Ok, I thought, perhaps naively, that since fairshare data is simply stored in a file called ‘usage’, the secondary could just copy that file over and use the information, or if it’s on a shared filesystem, then it’s available to the secondary as it’s only usage file. So, even if the secondary doesn’t contact primary’s scheduler and always spawns its own scheduler, it can just use the fairshare data that was already there. If this is vaid, then approach 2 would be as good as approach 1 for fairshare. If it’s not valid and approach 1 is far better than approach 2 for fairshare, then ya we should go with 1. ---
**Author:** ![mkaro](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/mkaro/32/85_2.png) [@mkaro](https://community.openpbs.org/u/mkaro)\ **Post date:** [April 4, 2018, 2:05pm UTC](https://community.openpbs.org/t/pp-832-which-scheduler-to-talk-to-while-taking-over-from-primary/889/8 "2018-04-04T14:05:05Z")
I favor approach #2. If the reason for failover is network “flakiness” then it’s possible packets are being dropped along the way. We would not want the secondary server relying on the primary scheduler which could lead to unexpected and confusing behavior. Best to send SCH\_QUIT to the primary scheduler and spawn a local instance, which is unlikely to be affected by network problems. Please keep in mind all of the code you are working with was written when RPP was used for communication. Perhaps the reason the secondary server attempts to contact the primary scheduler only once could have something to do with how RPP behaved. We may want to do things differently now that TPP is our primary means of communication. ---
**Author:** ![prakashcv13](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/prakashcv13/32/65_2.png) [@prakashcv13](https://community.openpbs.org/u/prakashcv13)\ **Post date:** [April 4, 2018, 4:23pm UTC](https://community.openpbs.org/t/pp-832-which-scheduler-to-talk-to-while-taking-over-from-primary/889/9 "2018-04-04T16:23:27Z")
> [@agrawalravi90](#): > > the secondary could just copy that file over and use the information, or if it’s on a shared filesystem, While I was focussing only on the issue at hand, I like the idea of having the usage file on the shared filesystem. @mkaro - Your reason has made me inclined towards solution 2, so I have updated it to have the “usage” file on the shared filesystem. ---
**Author:** ![agrawalravi90](https://avatars.discourse-cdn.com/v4/letter/a/848f3c/32.png) [@agrawalravi90](https://community.openpbs.org/u/agrawalravi90)\ **Post date:** [April 4, 2018, 4:59pm UTC](https://community.openpbs.org/t/pp-832-which-scheduler-to-talk-to-while-taking-over-from-primary/889/10 "2018-04-04T16:59:58Z")
I’d also like to get @bhroam’s inputs on this. ---
**Author:** ![bhroam](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bhroam/32/20_2.png) [@bhroam](https://community.openpbs.org/u/bhroam)\ **Post date:** [April 4, 2018, 8:54pm UTC](https://community.openpbs.org/t/pp-832-which-scheduler-to-talk-to-while-taking-over-from-primary/889/11 "2018-04-04T20:54:28Z")
@prakashcv13 is correct. The reason we have the secondary talk with the primary is for fairshare reasons. If we make a switch from one scheduler to another, we will lose some amount of fairshare data. The amount is one cycle’s worth. At the time we made this decision, it made sense to try and talk with the primary. A conversation about this topic with @prakashcv13 and @subhasisb changed my mind on this subject. Failover is an exception to the rule. It’s a very complex exception. We’re talking about having PBS switch from one host to another and keep running smoothly. Any added complexity we add to this system is another way everything can fail. I now look at having the primary talk to the secondary as added complexity. Option two has the secondary tell the primary to quit. We’re once again in an exceptional case here. If the reason we’re failing over is due to network issues between the primary and the secondary, the primary might miss being told to quit. We are then in the worse case scenario. The primary is still up and running with a view of the fairshare usage. The secondary takes over and runs for a while. When the primary takes back over, the primary scheduler’s stale view of the usage takes back over. We lose all usage accumulated while the secondary was up. I vote for a hybrid between one and two. Option one has the added complexity of having the secondary talk with the primary. Option two has the problem that the primary might not receive the signal to quit. I suggest the secondary ignores the primary and starts up its own scheduler. When the primary takes back over, it will tell the primary scheduler to reread the usage data. I’m not sure you want the init script to always restart the scheduler. There are times when only one of the daemons is down and admin will use the init script to start it. Since the other daemons are up, they are ignored. If we always restart the scheduler, we will be creating a situation where we will be losing fairshare data when we don’t need to. There is another way around this. The server runs a special cycle when it initially comes up. It’s called SCH\_SCHEDULE\_FIRST. The scheduler can reread the fairshare usage on this special cycle. Bhroam ---
**Author:** ![prakashcv13](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/prakashcv13/32/65_2.png) [@prakashcv13](https://community.openpbs.org/u/prakashcv13)\ **Post date:** [April 9, 2018, 9:02am UTC](https://community.openpbs.org/t/pp-832-which-scheduler-to-talk-to-while-taking-over-from-primary/889/12 "2018-04-09T09:02:55Z")
@bhroam - thank you for your inputs. How about we include @agrawalravi90’s suggestion to have “usage” file in a shared location to the hybrid solution you are suggesting? ---
**Author:** ![bhroam](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bhroam/32/20_2.png) [@bhroam](https://community.openpbs.org/u/bhroam)\ **Post date:** [April 9, 2018, 6:09pm UTC](https://community.openpbs.org/t/pp-832-which-scheduler-to-talk-to-while-taking-over-from-primary/889/13 "2018-04-09T18:09:21Z")
@prakashcv13 failover works with a shared PBS\_HOME. The usage file is already stored there. There is no need to change anything to make it work 🙂 Bhroam ---
**Author:** ![prakashcv13](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/prakashcv13/32/65_2.png) [@prakashcv13](https://community.openpbs.org/u/prakashcv13)\ **Post date:** [April 9, 2018, 9:36pm UTC](https://community.openpbs.org/t/pp-832-which-scheduler-to-talk-to-while-taking-over-from-primary/889/14 "2018-04-09T21:36:06Z")
:). I should have looked at the code before asking the question.If that is the case, we should also make the scheduler on secondary read the usage at the time of taking over by explicitly sending a SCH\_SCHEDULE\_FIRST. I have updated the [design](https://pbspro.atlassian.net/wiki/spaces/PD/pages/261193738/PP-832+PBSPro+failover+secondary+server+fails+to+continuously+check+whether+it+needs+to+start+a+scheduler+locally) with a single solution now, please let me know your thoughts. Thanks, Prakash ---
**Author:** ![bhroam](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bhroam/32/20_2.png) [@bhroam](https://community.openpbs.org/u/bhroam)\ **Post date:** [April 9, 2018, 9:58pm UTC](https://community.openpbs.org/t/pp-832-which-scheduler-to-talk-to-while-taking-over-from-primary/889/15 "2018-04-09T21:58:40Z")
The first time the secondary takes over, we’ll be starting the secondary scheduler, so no problems there. The scheduler will read the usage file when it starts up. The question is what happens to the secondary scheduler when the primary server takes back over? If we take the secondary scheduler down, then we won’t have a problem. It’ll be started again when the secondary takes over again. If we leave the secondary scheduler up, we’ll need to do what you said. It’ll need to reread the usage. I read your new design. I just wanted to be clear that there is work to be done to make the scheduler reread its usage on a SCH\_SCHEDULE\_FIRST. It doesn’t reread the usage now. I don’t know if you want to make that more clear in your solution. Bhroam ---
**Author:** ![prakashcv13](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/prakashcv13/32/65_2.png) [@prakashcv13](https://community.openpbs.org/u/prakashcv13)\ **Post date:** [April 9, 2018, 10:02pm UTC](https://community.openpbs.org/t/pp-832-which-scheduler-to-talk-to-while-taking-over-from-primary/889/16 "2018-04-09T22:02:43Z")
Thank you @bhroam, I have updated the design to reflect that we need to make scheduler re-read usage data on receiving SCH\_SCHEDULE\_FIRST. ---
**Author:** ![bhroam](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bhroam/32/20_2.png) [@bhroam](https://community.openpbs.org/u/bhroam)\ **Post date:** [April 10, 2018, 12:13am UTC](https://community.openpbs.org/t/pp-832-which-scheduler-to-talk-to-while-taking-over-from-primary/889/17 "2018-04-10T00:13:10Z")
@prakashcv13 thanks for making the changes to your document. We should probably decide on the fate of the secondary scheduler when the primary server takes back over. Right now your solution is silent on that. Do you want to tell the scheduler to quit? ---
**Author:** ![prakashcv13](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/prakashcv13/32/65_2.png) [@prakashcv13](https://community.openpbs.org/u/prakashcv13)\ **Post date:** [April 10, 2018, 11:20am UTC](https://community.openpbs.org/t/pp-832-which-scheduler-to-talk-to-while-taking-over-from-primary/889/18 "2018-04-10T11:20:06Z")
@bhroam, the current behavior itself is that the scheduler on the secondary goes away when the primary comes up. ---
**Author:** ![bhroam](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bhroam/32/20_2.png) [@bhroam](https://community.openpbs.org/u/bhroam)\ **Post date:** [April 10, 2018, 5:54pm UTC](https://community.openpbs.org/t/pp-832-which-scheduler-to-talk-to-while-taking-over-from-primary/889/19 "2018-04-10T17:54:55Z")
I never knew that. It sounds like the right thing to do. ---
**Author:** ![agrawalravi90](https://avatars.discourse-cdn.com/v4/letter/a/848f3c/32.png) [@agrawalravi90](https://community.openpbs.org/u/agrawalravi90)\ **Post date:** [April 11, 2018, 12:10am UTC](https://community.openpbs.org/t/pp-832-which-scheduler-to-talk-to-while-taking-over-from-primary/889/20 "2018-04-11T00:10:14Z")
Hey Prakash, Just clarifying something, the doc mentions that the primary will issue SCH\_SCHEDULE\_FIRST when it comes back up, don’t we need to do SCH\_CONFIGURE instead? That’s where it frees conf.fairshare and calls schedinit(), which reads the usage file to recreate conf.fairshare. SCH\_SCHEDULE\_FIRST might also lead to that, it just doesn’t seem obvious from the code, so I wanted to clarify it once. Thanks, Ravi [Next page](https://community.openpbs.org/t/pp-832-which-scheduler-to-talk-to-while-taking-over-from-primary/889.md?page=2)