# PP-339 and PP-647:release vnodes early from running jobs **URL:** \ **Category:** Developers\ **Created:** [January 24, 2017, 9:04pm UTC](https://community.openpbs.org/t/pp-339-and-pp-647-release-vnodes-early-from-running-jobs/419 "2017-01-24T21:04:55Z")\ **Posts on this page:** 20\ **Page:** 3
**Author:** ![bhroam](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bhroam/32/20_2.png) [@bhroam](https://community.openpbs.org/u/bhroam)\ **Post date:** [March 1, 2017, 10:44pm UTC](https://community.openpbs.org/t/pp-339-and-pp-647-release-vnodes-early-from-running-jobs/419/41 "2017-03-01T22:44:41Z")
Hey Al, Thanks for making the updates. My only comment is to do with the API call. I don’t think we have any other call that takes a space delimited format. I’d rather not add one if possible. I see two possible alternitives. qstat can take multiple job ids. It takes a comma delimited list. The second is to take the format given to qrun -H. It’s a + delimited list of vnodes. What do you think? Bhroam ---
**Author:** ![bayucan](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bayucan/32/24_2.png) [@bayucan](https://community.openpbs.org/u/bayucan)\ **Post date:** [March 1, 2017, 11:49pm UTC](https://community.openpbs.org/t/pp-339-and-pp-647-release-vnodes-early-from-running-jobs/419/42 "2017-03-01T23:49:23Z")
I can do the plus (+) separated list of vnodes. ---
**Author:** ![scc](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/scc/32/234_2.png) [@scc](https://community.openpbs.org/u/scc)\ **Post date:** [March 2, 2017, 6:32pm UTC](https://community.openpbs.org/t/pp-339-and-pp-647-release-vnodes-early-from-running-jobs/419/43 "2017-03-02T18:32:04Z")
Thanks Al, the changes look good. 1. In the event that a node is released from a job early, it is good that the EDD now states that the execjob\_end event will be triggered, but it does not explicitly state that the execjob\_end hook would never run on such a host, which is different than what the current documentation says about execjob\_end hooks, and so I think needs to be specifically called out in the EDD. 2. In interface 2 I do not see that this comment was addressed: “this will do an equivalent of ‘pbs\_release\_nodes -a’ for releasing all the sister vnodes when…” I believe that should say “for releasing all the sister nodes when…”, in keeping with the -a functionality. 1. There are 2 items both labeled as “Interface 5”, they both relate to accounting log changes, but I do not think the duplicate interface numbers are intentional? ---
**Author:** ![bayucan](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bayucan/32/24_2.png) [@bayucan](https://community.openpbs.org/u/bayucan)\ **Post date:** [March 3, 2017, 6:34am UTC](https://community.openpbs.org/t/pp-339-and-pp-647-release-vnodes-early-from-running-jobs/419/44 "2017-03-03T06:34:16Z")
> [@scc](#): > > 1. In the event that a node is released from a job early, it is good that the EDD now states that the execjob\_end event will be triggered, but it does not explicitly state that the execjob\_end hook would never run on such a host, which is different than what the current documentation says about execjob\_end hooks, and so I think needs to be specifically called out in the EDD Good point. I’ll add a note to say that in this case, the execjob\_end hook will not execute. > [@scc](#): > > 1. In interface 2 I do not see that this comment was addressed: > > “this will do an equivalent of ‘pbs\_release\_nodes -a’ for releasing all the sister vnodes when…” I believe that should say “for releasing all the sister nodes when…”, in keeping with the -a functionality. I must have missed addressing this. I’ll update. > [@scc](#): > > 1. There are 2 items both labeled as “Interface 5”, they both relate to accounting log changes, but I do not think the duplicate interface numbers are intentional? It should not be duplicated. I’ll fix. You can take a look at v15 now of the EDD where I made the changes. ---
**Author:** ![scc](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/scc/32/234_2.png) [@scc](https://community.openpbs.org/u/scc)\ **Post date:** [March 3, 2017, 2:54pm UTC](https://community.openpbs.org/t/pp-339-and-pp-647-release-vnodes-early-from-running-jobs/419/45 "2017-03-03T14:54:17Z")
Thanks Al! I have no further comments, looks great to me. ---
**Author:** ![jon](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/jon/32/23_2.png) [@jon](https://community.openpbs.org/u/jon)\ **Post date:** [March 3, 2017, 3:43pm UTC](https://community.openpbs.org/t/pp-339-and-pp-647-release-vnodes-early-from-running-jobs/419/46 "2017-03-03T15:43:36Z")
> [@bhroam](#): > > I don’t think we have any other call that takes a space delimited format. I’d rather not add one if possible. Maybe I am missing something but I believe that we already use space delimited format. We use it for qstat, qdel, and pbsnodes cli. I would rather not use the “+” delimiter from the cli. Now if this is only for the internal API calls then please disregard my comment. ---
**Author:** ![bayucan](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bayucan/32/24_2.png) [@bayucan](https://community.openpbs.org/u/bayucan)\ **Post date:** [March 3, 2017, 4:51pm UTC](https://community.openpbs.org/t/pp-339-and-pp-647-release-vnodes-early-from-running-jobs/419/47 "2017-03-03T16:51:09Z")
Ir’s only for the API call, Jon. ---
**Author:** ![gmatthew](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/gmatthew/32/32_2.png) [@gmatthew](https://community.openpbs.org/u/gmatthew)\ **Post date:** [March 14, 2017, 5:26am UTC](https://community.openpbs.org/t/pp-339-and-pp-647-release-vnodes-early-from-running-jobs/419/48 "2017-03-14T05:26:10Z")
Bob Ciotti has chimed in on 2 additional types of action for pbs\_release\_nodes he’d like to see: 1. Support specifying the number of nodes that should be kept. 2. Support specifying a select statement that is a subset of the job in its current form. These provide a more natural expression of what to keep in the job. The select statement, in particular, will allow users to not have to play games w/ parsing PBS\_NODEFILE or job attributes to know which nodes they want to keep or toss. ---
**Author:** ![bayucan](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bayucan/32/24_2.png) [@bayucan](https://community.openpbs.org/u/bayucan)\ **Post date:** [March 14, 2017, 4:28pm UTC](https://community.openpbs.org/t/pp-339-and-pp-647-release-vnodes-early-from-running-jobs/419/49 "2017-03-14T16:28:29Z")
> [@gmatthew](#): > > Bob Ciotti has chimed in on 2 additional types of action for pbs\_release\_nodes he’d like to see: > > Support specifying the number of nodes that should be kept. Interesting…so something like pbs\_release\_nodes -k \<‘number\_of\_sister\_nodes’\> which requests the release of \<‘number of nodes’\> - 1 sister nodes. I’ll have to check to make sure \<‘number\_of\_nodes’\> is not more than the actual number of nodes assigned to the job not including the mother superior node. > [@gmatthew](#): > > Support specifying a select statement that is a subset of the job in its current form. > > These provide a more natural expression of what to keep in the job. The select statement, in particular, will allow users to not have to play games w/ parsing PBS\_NODEFILE or job attributes to know which nodes they want to keep or toss. I’m confused with the ‘select’ part. Can you give me an example? ---
**Author:** ![gmatthew](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/gmatthew/32/32_2.png) [@gmatthew](https://community.openpbs.org/u/gmatthew)\ **Post date:** [March 14, 2017, 5:37pm UTC](https://community.openpbs.org/t/pp-339-and-pp-647-release-vnodes-early-from-running-jobs/419/50 "2017-03-14T17:37:53Z")
> [@bayucan](#): > > > [@gmatthew](#): > > > > Support specifying a select statement that is a subset of the job in its current form. > > > > These provide a more natural expression of what to keep in the job. The select statement, in particular, will allow users to not have to play games w/ parsing PBS\_NODEFILE or job attributes to know which nodes they want to keep or toss. > > I’m confused with the ‘select’ part. Can you give me an example? Suppose I get a job by asking for: qsub -l select=15:model=abc:mpiprocs=5+4:model=abc:bigmem=true:mpiprocs=1+30:model=def:mpiprocs=32 Then I want to release most of the nodes, but keep 2 of the non-bigmem abc nodes, and 2 of the bigmem abc nodes. It’d be much easier for the user to provide a new select statement: pbs\_release\_nodes -k select=2:model=abc:mpiprocs=5+2:model=abc:bigmem=true:mpiprocs=1 than to have to determine via PBS\_NODEFILE/qstat/pbsnodes which nodes have to be individually specified to pare back the job to that smaller select statement description. ---
**Author:** ![bayucan](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bayucan/32/24_2.png) [@bayucan](https://community.openpbs.org/u/bayucan)\ **Post date:** [March 14, 2017, 7:51pm UTC](https://community.openpbs.org/t/pp-339-and-pp-647-release-vnodes-early-from-running-jobs/419/51 "2017-03-14T19:51:35Z")
> [@gmatthew](#): > > Suppose I get a job by asking for: > > qsub -l select=15:model=abc:mpiprocs=5+4:model=abc:bigmem=true:mpiprocs=1+30:model=def:mpiprocs=32 > > Then I want to release most of the nodes, but keep 2 of the non-bigmem abc nodes, and 2 of the bigmem abc nodes. It’d be much easier for the user to provide a new select statement: > > pbs\_release\_nodes -k select=2:model=abc:mpiprocs=5+2:model=abc:bigmem=true:mpiprocs=1 > > than to have to determine via PBS\_NODEFILE/qstat/pbsnodes which nodes have to be individually specified to pare back the job to that smaller select statement descripti I see, now I understand. I can see how this is a good addition to release nodes feature. ---
**Author:** ![scc](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/scc/32/234_2.png) [@scc](https://community.openpbs.org/u/scc)\ **Post date:** [March 17, 2017, 3:10pm UTC](https://community.openpbs.org/t/pp-339-and-pp-647-release-vnodes-early-from-running-jobs/419/52 "2017-03-17T15:10:50Z")
Thanks Greg and Al, I have filed PP-724 and PP-725 to capture these new user stories. Do we all agree that these proposed capabilities should be looked at as possible future enhancements, rather than extending the work currently being done for PP-339 and PP-647? ---
**Author:** ![bayucan](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bayucan/32/24_2.png) [@bayucan](https://community.openpbs.org/u/bayucan)\ **Post date:** [March 17, 2017, 3:52pm UTC](https://community.openpbs.org/t/pp-339-and-pp-647-release-vnodes-early-from-running-jobs/419/53 "2017-03-17T15:52:10Z")
> [@scc](#): > > Thanks Greg and Al, I have filed PP-724 and PP-725 to capture these new user stories. Do we all agree that these proposed capabilities should be looked at as possible future enhancements, rather than extending the work currently being done for PP-339 and PP-647? It’s fine with me… ---
**Author:** ![gmatthew](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/gmatthew/32/32_2.png) [@gmatthew](https://community.openpbs.org/u/gmatthew)\ **Post date:** [March 21, 2017, 8:01pm UTC](https://community.openpbs.org/t/pp-339-and-pp-647-release-vnodes-early-from-running-jobs/419/54 "2017-03-21T20:01:41Z")
> [@scc](#): > > Thanks Greg and Al, I have filed PP-724 and PP-725 to capture these new user stories. Do we all agree that these proposed capabilities should be looked at as possible future enhancements, rather than extending the work currently being done for PP-339 and PP-647? Let’s assume for now they are future enhancements, and we can clarify with Ciotti at the next Altair/NAS meeting. ---
**Author:** ![bayucan](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bayucan/32/24_2.png) [@bayucan](https://community.openpbs.org/u/bayucan)\ **Post date:** [April 7, 2017, 9:17pm UTC](https://community.openpbs.org/t/pp-339-and-pp-647-release-vnodes-early-from-running-jobs/419/55 "2017-04-07T21:17:01Z")
I just updated the node ramp down design document. It’s now v18.You might want to go to “Page History” and compare v.15 against v.18 (current). It just contains some minor update to output of some examples, also the error message to doing pbs\_release\_nodes of a cpuset node or a Cray XC node is changed slightly to be more consistent with format in other commands. Finally, I’ve added a new private, experimental server attribute to allow showing in qstat -f the values of internal attributes created by PBS server in order to implement this feature. The option is just an aid in debugging PBS. [Node Ramp Down Design](https://pbspro.atlassian.net/wiki/display/PD/PP-339+and+PP-647%3A+node+ramp+down+feature%3A+release+vnodes+early+from+running+jobs) ---
**Author:** ![billnitzberg](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/billnitzberg/32/6_2.png) [@billnitzberg](https://community.openpbs.org/u/billnitzberg)\ **Post date:** [April 13, 2017, 7:22pm UTC](https://community.openpbs.org/t/pp-339-and-pp-647-release-vnodes-early-from-running-jobs/419/56 "2017-04-13T19:22:36Z")
Looking to the future, we are likely to want to support adding resources to a job (in addition to releasing resources). To extend PBS Pro to add resources to jobs (in the future), it would be valuable to design this feature (today) to be easily extendable (syntax and accounting) and to fit naturally with that future direction. For the “add resources” CLI/API, the natural opposite of pbs\_release\_nodes would be pbs\_add\_nodes, but that seems very unnatural and un-PBS-y. A better choice would be to use the existing PBS resource request language (-l select=XYZ) to either: - add resources, e.g., pbs\_add\_resources -l select=XYZ … - or replace the job’s resources, e.g., pbs\_resize\_resources -l selectXYZ" … The latter obviously could be used for either adding or releasing, and has the design advantages of: (a) using the existing PBS syntax, (b) having PBS handle the calculation of which chunks are on which nodes, and © naturally supporting both node-level and job-wide resources (e.g., expensive software licenses). (Note: “pbs\_release\_nodes -a” does have all the above design advantages.) I particularly don’t like the idea of forcing the user to do work that could be automated by PBS, so (b) strikes me as particularly onerous. (Allowing node names as an option is good; having them as the only option is not so nice.) Even more than the CLI/API, I feel it is really important to ensure the accounting records will handle the add case (in addition to the release case), as the whole PBS Pro ecosystem builds tools that depend on and consume the accounting records. Here I am not not sure whether it would work or not… would the proposed new u, c, and e records also work for adding/resizing in general? Would we need any changes in them to support adding resources? Thanks! ---
**Author:** ![bayucan](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bayucan/32/24_2.png) [@bayucan](https://community.openpbs.org/u/bayucan)\ **Post date:** [April 24, 2017, 10:49pm UTC](https://community.openpbs.org/t/pp-339-and-pp-647-release-vnodes-early-from-running-jobs/419/57 "2017-04-24T22:49:39Z")
> [@billnitzberg](#): > > Looking to the future, we are likely to want to support adding resources to a job (in addition to releasing resources). To extend PBS Pro to add resources to jobs (in the future), it would be valuable to design this feature (today) to be easily extendable (syntax and accounting) and to fit naturally with that future direction. > > For the “add resources” CLI/API, the natural opposite of pbs\_release\_nodes would be pbs\_add\_nodes, but that seems very unnatural and un-PBS-y. A better choice would be to use the existing PBS resource request language (-l select=XYZ) to either: > > add resources, e.g., pbs\_add\_resources -l select=XYZ … > or replace the job’s resources, e.g., pbs\_resize\_resources -l selectXYZ" … > > The latter obviously could be used for either adding or releasing, and has the design advantages of: > (a) using the existing PBS syntax, > (b) having PBS handle the calculation of which chunks are on which nodes, and > (c) naturally supporting both node-level and job-wide resources (e.g., expensive software licenses). > (Note: “pbs\_release\_nodes -a” does have all the above design advantages.) > > I particularly don’t like the idea of forcing the user to do work that could be automated by PBS, so (b) strikes me as particularly onerous. (Allowing node names as an option is good; having them as the only option is not so nice.) > > Even more than the CLI/API, I feel it is really important to ensure the accounting records will handle the add case (in addition to the release case), as the whole PBS Pro ecosystem builds tools that depend on and consume the accounting records. Here I am not not sure whether it would work or not… would the proposed new u, c, and e records also work for adding/resizing in general? Would we need any changes in them to support adding resources? > > Thanks! I believe the new u, c, and e records could easily be extended when dealing with adding nodes. Each record can reflect the exact exec\_vnode, exec\_host, resources\_used.\* values consistent with the action, whether the nodes are added or nodes released. In regards to using pbs\_resize\_nodes as opposed to pbs\_release\_nodes, I’m not sure if we should change the command name now. I know we have ticket filed to eventually support -l select syntax in pbs\_release\_nodes, and maybe can just add to that ticket to also rename “pbs\_release\_nodes” to “pbs\_resize\_nodes” when we support that… ---
**Author:** ![lisa-altair](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/lisa-altair/32/643_2.png) [@lisa-altair](https://community.openpbs.org/u/lisa-altair)\ **Post date:** [June 21, 2017, 2:52am UTC](https://community.openpbs.org/t/pp-339-and-pp-647-release-vnodes-early-from-running-jobs/419/58 "2017-06-21T02:52:45Z")
Hi @bayucan I’m sorry to be chiming in so late, but @bhroam asked me some questions and pointed me to a portion of your code, which lead me to your external design…and well, here I am with questions. 1. Can you please add a link from your external design to this forum discussion? 2. the design says "pbs\_release\_nodes is not currently supported with nodes/vnodes that are tied to Cray XC systems, as the ALPS reservation cannot be modified right now. " It isn’t really clear what “tied to Cray XC systems” means. But then in looking at the code it looks like you plan to check for CRAY\_LOGIN and CRAY\_COMPUTE. But Cray login nodes do not have an ALPS reservation…if a job had requested the resources on more than 1 login node, theoretically all but one login node could also be freed. I understand that you might not want to allow it anyway, but I thought I’d point it out so you could clarify. 3. PBS can manage a combination of linux systems together with a Cray X\* series system. Would you expect those linux systems to be able to participate in node ramp down? 3a) If so, those linux systems can potentially be labeled by the admin with a vntype of “cray\_compile” (in fact it’s an example in our guides) In which case @bhroam code suggestion of strstr for “cray” won’t work 3b) If not, how will you prevent those associated linux systems from participating in node ramp down since their vntype won’t be set to either cray\_login or cray\_compute and might not be set to “cray” anything. 4. PBS supports more than the XC Cray system. You should probably update the design and code to state “Cray X\* series” in all the places it says XC. ---
**Author:** ![bayucan](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bayucan/32/24_2.png) [@bayucan](https://community.openpbs.org/u/bayucan)\ **Post date:** [June 21, 2017, 4:33pm UTC](https://community.openpbs.org/t/pp-339-and-pp-647-release-vnodes-early-from-running-jobs/419/59 "2017-06-21T16:33:26Z")
> [@lisa-altair](#): > > Hi @bayucan I’m sorry to be chiming in so late, but @bhroam asked me some questions and pointed me to a portion of your code, which lead me to your external design…and well, here I am with questions. > > 1. Can you please add a link from your external design to this forum discussion? I’ll do. > [@lisa-altair](#): > > 1. the design says "pbs\_release\_nodes is not currently supported with nodes/vnodes that are tied to Cray XC systems, as the ALPS reservation cannot be modified right now. " > It isn’t really clear what “tied to Cray XC systems” means. But then in looking at the code it looks like you plan to check for CRAY\_LOGIN and CRAY\_COMPUTE. But Cray login nodes do not have an ALPS reservation…if a job had requested the resources on more than 1 login node, theoretically all but one login node could also be freed. I understand that you might not want to allow it anyway, but I thought I’d point it out so you could clarify. Thanks for clarifying this. I realized now that only cray\_compute nodes are involved in alps reservation, but I think it’s still a good idea to not allow cray\_login nodes to be ramped down. I just have to update the EDD to specifically say vntypes of “cray\_login” and “cray\_compute” are not allowed to be released early. > [@lisa-altair](#): > > 1. PBS can manage a combination of linux systems together with a Cray X\* series system. Would you expect those linux systems to be able to participate in node ramp down? Yes. > [@lisa-altair](#): > > 3a) If so, those linux systems can potentially be labeled by the admin with a vntype of “cray\_compile” (in fact it’s an example in our guides) In which case @bhroam code suggestion of strstr for “cray” won’t work That’s why I did not want to do strstr of “cray” in the first place. > [@lisa-altair](#): > > 3b) If not, how will you prevent those associated linux systems from participating in node ramp down since their vntype won’t be set to either cray\_login or cray\_compute and might not be set to “cray” anything. > 4) PBS supports more than the XC Cray system. You should probably update the design and code to state “Cray X\* series” in all the places it says XC. I’ll update the EDD also to say “Cray X\*series” instead of XC. ---
**Author:** ![lisa-altair](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/lisa-altair/32/643_2.png) [@lisa-altair](https://community.openpbs.org/u/lisa-altair)\ **Post date:** [June 22, 2017, 11:04pm UTC](https://community.openpbs.org/t/pp-339-and-pp-647-release-vnodes-early-from-running-jobs/419/60 "2017-06-22T23:04:04Z")
> [@bayucan](#): > > That’s why I did not want to do strstr of “cray” in the first place. I understand that. However @bhroam still has a good point, we know that there will be another vntype for Cray X\* series in the near future. I can easily see all of us missing the fact that the new vntype is not added to the checks of what is not allowed to do node ramp down, and who knows what would happen then… [Previous page](https://community.openpbs.org/t/pp-339-and-pp-647-release-vnodes-early-from-running-jobs/419.md?page=2) [Next page](https://community.openpbs.org/t/pp-339-and-pp-647-release-vnodes-early-from-running-jobs/419.md?page=4)