# PP-305: If server\_dyn\_res script does not return , Scheduler hangs **URL:** \ **Category:** Developers\ **Created:** [May 17, 2017, 9:59am UTC](https://community.openpbs.org/t/pp-305-if-server-dyn-res-script-does-not-return-scheduler-hangs/547 "2017-05-17T09:59:27Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![varunsonkar](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/varunsonkar/32/114_2.png) [@varunsonkar](https://community.openpbs.org/u/varunsonkar)\ **Post date:** [May 17, 2017, 9:59am UTC](https://community.openpbs.org/t/pp-305-if-server-dyn-res-script-does-not-return-scheduler-hangs/547/1 "2017-05-17T09:59:27Z")
Hi All, As of today, if the server\_dyn\_res program/script does not return or hangs. The scheduler keeps on waiting for the script to complete the execution. Following is the design document for the proposed solution to this hang issue: _**[Design Document](https://pbspro.atlassian.net/wiki/pages/viewpage.action?pageId=51975257)**_ Please review the proposed design and provide the comments/feedback for same. Regards, Varun ---
**Author:** ![scc](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/scc/32/234_2.png) [@scc](https://community.openpbs.org/u/scc)\ **Post date:** [May 17, 2017, 1:57pm UTC](https://community.openpbs.org/t/pp-305-if-server-dyn-res-script-does-not-return-scheduler-hangs/547/2 "2017-05-17T13:57:58Z")
Hi Varun, a few comments: 1. Currently the EDD is written to sound like there is only ever 1 server\_dyn\_res script (“THE server\_dyn\_res” is written twice, but I think it should be “A server\_dyn\_res”). 2. Related to 1), the EDD should be explicit about whether this timeout applies to ALL server\_dyn\_res scripts together, or EACH server\_dyn\_res individually. 3. The EDD should be explicit about how the resources related to a timed out script run is treated in the cycle in which it timed out. Does the cycle continue normally but the value of that resource is assumed to be 0? 4. Related to 3), assuming the timed out resource value is treated as 0 for that cycle, the log message in interface 2 should be explicit about this. That is not necessary, though, if in this scenario the log will contain something like these current messages in addition to the new message in interface 2: 05/17/2017 09:54:07;0080;pbs\_sched;Svr;server\_dyn\_res;Error piping to program /bin/get\_foo. 05/17/2017 09:54:07;0100;pbs\_sched;Svr;server\_dyn\_res;/bin/get\_foo = 0 If those messages will be printed upon timeout in addition to the new one in interface 2 I think the EDD should be explicit about it. 1. I am not sure I like the name in interface 1. I like the simpler “server\_dyn\_res\_timeout” better, or maybe “server\_dyn\_res\_alarm”. Did you insert the “prog” to try to make it clearer that the timeout applies to each individual script/program? I think if the EDD and docs are explicit about this then the simpler attribute name is better. Thanks! ---
**Author:** ![jon](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/jon/32/23_2.png) [@jon](https://community.openpbs.org/u/jon)\ **Post date:** [May 17, 2017, 10:50pm UTC](https://community.openpbs.org/t/pp-305-if-server-dyn-res-script-does-not-return-scheduler-hangs/547/3 "2017-05-17T22:50:20Z")
> [@scc](#): > > 1. Related to 1), the EDD should be explicit about whether this timeout applies to ALL server\_dyn\_res scripts together, or EACH server\_dyn\_res individually. I think that it should apply to each server\_dyn\_res to match the same behavior as the alarm in hooks > [@scc](#): > > 1. The EDD should be explicit about how the resources related to a timed out script run is treated in the cycle in which it timed out. Does the cycle continue normally but the value of that resource is assumed to be 0? This is a good questions. Sites use server\_dyn\_res in various ways. Some sites use it get license counts, other use it to alter jobs, etc. I think that we should assume zero and continue. If the site wants different behavior they will need to add a timeout in their script. > [@scc](#): > > 1. I am not sure I like the name in interface 1. I like the simpler “server\_dyn\_res\_timeout” better, or maybe “server\_dyn\_res\_alarm”. Did you insert the “prog” to try to make it clearer that the timeout applies to each individual script/program? I think if the EDD and docs are explicit about this then the simpler attribute name is better. I prefer server\_dyn\_res\_alarm ---
**Author:** ![jon](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/jon/32/23_2.png) [@jon](https://community.openpbs.org/u/jon)\ **Post date:** [May 17, 2017, 10:51pm UTC](https://community.openpbs.org/t/pp-305-if-server-dyn-res-script-does-not-return-scheduler-hangs/547/4 "2017-05-17T22:51:48Z")
In interface 1 I believe that we should set the default value to 30 sec to match the same default as we have in hooks. ---
**Author:** ![varunsonkar](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/varunsonkar/32/114_2.png) [@varunsonkar](https://community.openpbs.org/u/varunsonkar)\ **Post date:** [May 18, 2017, 9:56am UTC](https://community.openpbs.org/t/pp-305-if-server-dyn-res-script-does-not-return-scheduler-hangs/547/5 "2017-05-18T09:56:23Z")
Hi @jon and @scc, Thanks for the comments. > [@jon](#): > > > [@scc](#): > > > > 1. Related to 1), the EDD should be explicit about whether this timeout applies to ALL server\_dyn\_res scripts together, or EACH server\_dyn\_res individually. > > I think that it should apply to each server\_dyn\_res to match the same behavior as the alarm in hooks Yes the timeout applies for each server\_dyn\_res script. Added the point in the EDD. > [@jon](#): > > > [@scc](#): > > > > 1. The EDD should be explicit about how the resources related to a timed out script run is treated in the cycle in which it timed out. Does the cycle continue normally but the value of that resource is assumed to be 0? > > This is a good questions. Sites use server\_dyn\_res in various ways. Some sites use it get license counts, other use it to alter jobs, etc. I think that we should assume zero and continue. If the site wants different behavior they will need to add a timeout in their script. Thanks for putting this point I missed this. I also agree we should assume the value to be zero and continue. Added the point for this in the EDD. > [@jon](#): > > > [@scc](#): > > > > 1. I am not sure I like the name in interface 1. I like the simpler “server\_dyn\_res\_timeout” better, or maybe “server\_dyn\_res\_alarm”. Did you insert the “prog” to try to make it clearer that the timeout applies to each individual script/program? I think if the EDD and docs are explicit about this then the simpler attribute name is better. > > I prefer server\_dyn\_res\_alarm Modified the name in the EDD to ""server\_dyn\_res\_alarm ---
**Author:** ![varunsonkar](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/varunsonkar/32/114_2.png) [@varunsonkar](https://community.openpbs.org/u/varunsonkar)\ **Post date:** [May 18, 2017, 10:04am UTC](https://community.openpbs.org/t/pp-305-if-server-dyn-res-script-does-not-return-scheduler-hangs/547/6 "2017-05-18T10:04:03Z")
> [@scc](#): > > 1. Related to 3), assuming the timed out resource value is treated as 0 for that cycle, the log message in interface 2 should be explicit about this. That is not necessary, though, if in this scenario the log will contain something like these current messages in addition to the new message in interface 2: > > 05/17/2017 09:54:07;0080;pbs\_sched;Svr;server\_dyn\_res;Error piping to program /bin/get\_foo. > 05/17/2017 09:54:07;0100;pbs\_sched;Svr;server\_dyn\_res;/bin/get\_foo = 0 > > If those messages will be printed upon timeout in addition to the new one in interface 2 I think the EDD should be explicit about it. I have modified the EDD with following messages: … …;0080;pbs\_sched;Svr;server\_dyn\_res;program /bin/get\_foo timed out … …;0100;pbs\_sched;Svr;server\_dyn\_res;/bin/get\_foo = 0 I have not added “Error piping to program” message as I think it may not be a piping error just a hang issue. We have specific condition for piping error in the code which will remain intact. It is quite possible the program is piping but may be there are delays/slowness/hangs and it times out. So I prefer keeping only “timed out” message. Please review the updated EDD. Thanks, Varun ---
**Author:** ![varunsonkar](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/varunsonkar/32/114_2.png) [@varunsonkar](https://community.openpbs.org/u/varunsonkar)\ **Post date:** [May 18, 2017, 10:05am UTC](https://community.openpbs.org/t/pp-305-if-server-dyn-res-script-does-not-return-scheduler-hangs/547/7 "2017-05-18T10:05:02Z")
> [@jon](#): > > In interface 1 I believe that we should set the default value to 30 sec to match the same default as we have in hooks. Modified the default value to 30 sec. Please review the updated EDD. Thanks, Varun ---
**Author:** ![jon](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/jon/32/23_2.png) [@jon](https://community.openpbs.org/u/jon)\ **Post date:** [May 18, 2017, 2:59pm UTC](https://community.openpbs.org/t/pp-305-if-server-dyn-res-script-does-not-return-scheduler-hangs/547/8 "2017-05-18T14:59:15Z")
The changes look fine. One suggestion, lets log interface 3 at the default logging level. ---
**Author:** ![bhroam](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bhroam/32/20_2.png) [@bhroam](https://community.openpbs.org/u/bhroam)\ **Post date:** [May 18, 2017, 5:56pm UTC](https://community.openpbs.org/t/pp-305-if-server-dyn-res-script-does-not-return-scheduler-hangs/547/9 "2017-05-18T17:56:31Z")
@varunsonkar unless the two log messages (interfaces 2/3) are required for automated testing, I would suggest making them Unstable. Log messages are something that should be able to change without 1 years notice. Bhroam ---
**Author:** ![varunsonkar](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/varunsonkar/32/114_2.png) [@varunsonkar](https://community.openpbs.org/u/varunsonkar)\ **Post date:** [May 19, 2017, 4:48am UTC](https://community.openpbs.org/t/pp-305-if-server-dyn-res-script-does-not-return-scheduler-hangs/547/10 "2017-05-19T04:48:48Z")
> [@jon](#): > > The changes look fine. One suggestion, lets log interface 3 at the default logging level. Modified the EDD. Please review. Thanks, Varun ---
**Author:** ![varunsonkar](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/varunsonkar/32/114_2.png) [@varunsonkar](https://community.openpbs.org/u/varunsonkar)\ **Post date:** [May 19, 2017, 4:51am UTC](https://community.openpbs.org/t/pp-305-if-server-dyn-res-script-does-not-return-scheduler-hangs/547/11 "2017-05-19T04:51:22Z")
> [@bhroam](#): > > @varunsonkar unless the two log messages (interfaces 2/3) are required for automated testing, I would suggest making them Unstable. Log messages are something that should be able to change without 1 years notice. > > Bhroam Hi @bhroam, Thanks for the input I have updated the EDD as per your suggestion. Made the interfaces(2 and 3) as Unstable. Regards, Varun ---
**Author:** ![jon](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/jon/32/23_2.png) [@jon](https://community.openpbs.org/u/jon)\ **Post date:** [May 31, 2017, 3:59am UTC](https://community.openpbs.org/t/pp-305-if-server-dyn-res-script-does-not-return-scheduler-hangs/547/12 "2017-05-31T03:59:20Z")
Thanks for the changes the EDD looks good. Since no one else has commented in the last 12 days I suggest we wait for one more day before to see if there are anymore comments before we end discussion and move forward. ---
**Author:** ![vstumpf](https://avatars.discourse-cdn.com/v4/letter/v/dbc845/32.png) [@vstumpf](https://community.openpbs.org/u/vstumpf)\ **Post date:** [January 15, 2020, 2:23am UTC](https://community.openpbs.org/t/pp-305-if-server-dyn-res-script-does-not-return-scheduler-hangs/547/13 "2020-01-15T02:23:32Z")
I have updated the design, if server\_dyn\_res\_alarm is 0, the scheduler will not timeout the scripts. This will effectively be the same as the previous behavior.