# PP-482: Non-destructive walltime **URL:** \ **Category:** Developers\ **Created:** [June 24, 2017, 12:23am UTC](https://community.openpbs.org/t/pp-482-non-destructive-walltime/587 "2017-06-24T00:23:23Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![bhroam](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bhroam/32/20_2.png) [@bhroam](https://community.openpbs.org/u/bhroam)\ **Post date:** [June 24, 2017, 12:23am UTC](https://community.openpbs.org/t/pp-482-non-destructive-walltime/587/1 "2017-06-24T00:23:23Z")
I’ve just written up an EDD for PP-482. This will implement a new version of the walltime resource. The new resource will allow the scheduler to use policies like top jobs without the danger of jobs being killed if they exceed their walltime. Please provide feedback: [Design Doc](https://pbspro.atlassian.net/wiki/display/PD/PP-482%3A+Non-destructive+Walltime) ---
**Author:** ![agrawalravi90](https://avatars.discourse-cdn.com/v4/letter/a/848f3c/32.png) [@agrawalravi90](https://community.openpbs.org/u/agrawalravi90)\ **Post date:** [June 26, 2017, 8:31am UTC](https://community.openpbs.org/t/pp-482-non-destructive-walltime/587/2 "2017-06-26T08:31:57Z")
Hey Bhroam, Thanks for the descriptive doc! Just a thought: Now that we have a way of classifying jobs via the job equivalence class code, will it useful to expose the job classes to admins so that they can set soft walltime on an entire class of jobs in one shot? About the doc, I just wanted to confirm that in the bullet: "If dedicated time is used, soft\_walltime will be used to see if the job will finish before dedicated time starts." you did mean to use soft-walltime for dedicated time windows right? (and not hard walltime) ---
**Author:** ![jon](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/jon/32/23_2.png) [@jon](https://community.openpbs.org/u/jon)\ **Post date:** [June 26, 2017, 5:55pm UTC](https://community.openpbs.org/t/pp-482-non-destructive-walltime/587/3 "2017-06-26T17:55:46Z")
Looks good. I just have a question and a few comments. In the document you mention that the default value for soft\_walltime\_extension\_factor is 1.0. Would it make sense to list this in interface 2? > [@agrawalravi90](#): > > Just a thought: Now that we have a way of classifying jobs via the job equivalence class code, will it useful to expose the job classes to admins so that they can set soft walltime on an entire class of jobs in one shot? This is an interesting idea. It would be useful if a site could provide information (i.e. a python dictionary or json file with the required information such that the scheduler could do a look up) and use that without having to have to use qalter or a hook to do it. However, for now I think we should start with just allowing an admin to set a default soft walltime for each queue or server. If they want more control then for now I would recommend setting it with a hook or qalter > [@agrawalravi90](#): > > you did mean to use soft-walltime for dedicated time windows right? (and not hard walltime) It makes sense to use the soft walltime for the dedicated time windows since a admin can set these without any guarantee that the jobs will be completed before the dedicated time is reached. Once it is reached then they have the option to handle the jobs as they deem necessary ---
**Author:** ![bhroam](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bhroam/32/20_2.png) [@bhroam](https://community.openpbs.org/u/bhroam)\ **Post date:** [June 26, 2017, 11:16pm UTC](https://community.openpbs.org/t/pp-482-non-destructive-walltime/587/4 "2017-06-26T23:16:50Z")
> [@agrawalravi90](#): > > Just a thought: Now that we have a way of classifying jobs via the job equivalence class code, will it useful to expose the job classes to admins so that they can set soft walltime on an entire class of jobs in one shot? The job equivalence class RFE was implemented in such a way that the admin doesn’t need to care about them. PBS determines the equivalence classes on its own. We don’t bother the admin about it. We can do a better job than the admin in any case. Doing what you suggest would give an external interface to the equivalence classes. I don’t think this is a good idea. Right now the equivalence classes can change from cycle to cycle depending on the site policies defined and the job mix. For example, in some cases the job’s euser attribute is used. In other cases it isn’t. It’ll be confusing the admin to understand what PBS is doing and why. I’d rather see the admins modify the jobs and then have PBS create equivalence classes from that. Admins know about jobs. They understand jobs. Teaching them how equivalence classes work and how they are created is not necessary. Let’s just let them deal with what they know. > [@agrawalravi90](#): > > About the doc, I just wanted to confirm that in the bullet: > “If dedicated time is used, soft\_walltime will be used to see if the job will finish before dedicated time starts.” > you did mean to use soft-walltime for dedicated time windows right? (and not hard walltime) The idea is that the admin is taking the machine under their control. If a job runs over into dedicated time, the admin can choose what to do with the job. If they requeue the job, there is no real difference if it hadn’t run in the first place. We might as well try and run the job in hopes it will end by its soft\_walltime. Bhroam ---
**Author:** ![billnitzberg](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/billnitzberg/32/6_2.png) [@billnitzberg](https://community.openpbs.org/u/billnitzberg)\ **Post date:** [June 28, 2017, 11:32pm UTC](https://community.openpbs.org/t/pp-482-non-destructive-walltime/587/5 "2017-06-28T23:32:10Z")
Happy to see the soft walltime idea finally getting some traction – thanks! Couple comments: 1. I suggest removing interface 2 (soft\_walltime\_extension\_factor) for several reasons: - Less complexity means less work (dev, QA, doc, training) and fewer bugs - Hopefully, admins will set the value of soft\_walltime so that it is rarely updated in practice - A feature like this can easily be added later (with no backward compatibility issues) - It doesn’t add much flexibility beyond "just doubling (up to “walltime”) - If something more complicated is needed (in a future release), it’s not clear that a simple, global, linear factor is the right solution; it would be better to collect feedback first, then implement the right solution. 1. PBS Pro already has “soft” used at the end of keywords, e.g., max\_run\_soft. Should this be soft\_walltime or walltime\_soft? I actually don’t have a strong feeling on this one, but wanted to bring it up because I know there is a strong desire for consistency in naming, and wanted others to comment (if desired). 2. Are there any interactions with min\_walltime and max\_walltime that need to be defined? Thanks again! ---
**Author:** ![jon](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/jon/32/23_2.png) [@jon](https://community.openpbs.org/u/jon)\ **Post date:** [June 29, 2017, 1:09am UTC](https://community.openpbs.org/t/pp-482-non-destructive-walltime/587/6 "2017-06-29T01:09:11Z")
> [@billnitzberg](#): > > I suggest removing interface 2 (soft\_walltime\_extension\_factor) for several reasons: > Less complexity means less work (dev, QA, doc, training) and fewer bugs > Hopefully, admins will set the value of soft\_walltime so that it is rarely updated in practice > A feature like this can easily be added later (with no backward compatibility issues) > It doesn’t add much flexibility beyond "just doubling (up to “walltime”) > If something more complicated is needed (in a future release), it’s not clear that a simple, global, linear factor is the right solution; it would be better to collect feedback first, then implement the right solution. I see you points here. And you are right that ideally this will not be used on very many jobs. The intent for this was to allow the admin to be able to adjust the soft\_walltime by a factor that makes sense to them depending on how aggressive they want to be when setting the soft\_walltime. As for how the factor is intended to work it would be an addition of the original requested walltime and not a doubling if set to 1. > [@billnitzberg](#): > > PBS Pro already has “soft” used at the end of keywords, e.g., max\_run\_soft. Should this be soft\_walltime or walltime\_soft? I actually don’t have a strong feeling on this one, but wanted to bring it up because I know there is a strong desire for consistency in naming, and wanted others to comment (if desired). For consistency with how we named queue/server limits it makes sense to call it walltime\_soft but that really doesn’t roll of the tongue. 🙂 However, since it is a job attribute and not a limit I would vote to leave it as is. > [@billnitzberg](#): > > Are there any interactions with min\_walltime and max\_walltime that need to be defined? Good point. I think the interactions need to be listed in the EDD ---
**Author:** ![bhroam](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bhroam/32/20_2.png) [@bhroam](https://community.openpbs.org/u/bhroam)\ **Post date:** [June 29, 2017, 1:56am UTC](https://community.openpbs.org/t/pp-482-non-destructive-walltime/587/7 "2017-06-29T01:56:24Z")
> [@billnitzberg](#): > > I suggest removing interface 2 (soft\_walltime\_extension\_factor) for several reasons: > Less complexity means less work (dev, QA, doc, training) and fewer bugs > Hopefully, admins will set the value of soft\_walltime so that it is rarely updated in practice > A feature like this can easily be added later (with no backward compatibility issues) > It doesn’t add much flexibility beyond "just doubling (up to “walltime”) > If something more complicated is needed (in a future release), it’s not clear that a simple, global, linear factor is the right solution; it would be better to collect feedback first, then implement the right solution. If we go on the assumption that the vast majority of jobs will only need to be extended a max of once, then I’m fine with this. We still need to define how an extension happens. I dislike just doubling the existing duration every time. I’d rather add the soft walltime on each time. If our assumption holds true, there is no difference. > [@billnitzberg](#): > > PBS Pro already has “soft” used at the end of keywords, e.g., max\_run\_soft. Should this be soft\_walltime or walltime\_soft? I actually don’t have a strong feeling on this one, but wanted to bring it up because I know there is a strong desire for consistency in naming, and wanted others to comment (if desired). > [@jon](#): > > For consistency with how we named queue/server limits it makes sense to call it walltime\_soft but that really doesn’t roll of the tongue. 🙂 However, since it is a job attribute and not a limit I would vote to leave it as is. I’m not so sure it’s most consistent to put it at the end. You can look at the fact that we only have soft at the end or that the vast majority of multi-word keywords read more like a sentence. Resources like min or max walltime or attributes like do\_not\_span\_psets are examples. I agree with Jon that we leave soft at the front. > [@billnitzberg](#): > > Are there any interactions with min\_walltime and max\_walltime that need to be defined? Yes, I need to update the EDD. STF jobs and soft walltime make no sense together. If you submit a job with a min walltime, you’re saying that this is the minimum amount of time that you need to run to get any real work done. The walltime can be set to the min walltime. You have to set your soft walltime smaller than your hard walltime. This means you’d have to set your soft walltime smaller than the minimum amount of time you need to get any real work done. Bhroam ---
**Author:** ![agurban](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/agurban/32/67_2.png) [@agurban](https://community.openpbs.org/u/agurban)\ **Post date:** [June 29, 2017, 3:20pm UTC](https://community.openpbs.org/t/pp-482-non-destructive-walltime/587/8 "2017-06-29T15:20:49Z")
I think we should try for better consistency with naming going forward. This would put “soft” at the end. It’s better to go with how we want things to work in the future. Yes, we have some wacky naming, but no need to keep that up. ---
**Author:** ![bhroam](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bhroam/32/20_2.png) [@bhroam](https://community.openpbs.org/u/bhroam)\ **Post date:** [June 29, 2017, 8:52pm UTC](https://community.openpbs.org/t/pp-482-non-destructive-walltime/587/9 "2017-06-29T20:52:26Z")
> [@agurban](#): > > I think we should try for better consistency with naming going forward. > This would put “soft” at the end. It’s better to go with how we want > things to work in the future. Yes, we have some wacky naming, but no > need to keep that up. Thanks for your comment, Anne. I agree that we should be consistent, but I don’t believe putting soft at the end is the most consistent thing we can do. I understand that the word soft currently only appears at the end. The vast majority of our multi-word resources and attributes read more like a sentence. I think it’d be more consistent to put soft first. It’d match min\_walltime and max\_walltime more closely. Following this logic, the limit attributes are the odd men out. Bhroam ---
**Author:** ![bhroam](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bhroam/32/20_2.png) [@bhroam](https://community.openpbs.org/u/bhroam)\ **Post date:** [June 29, 2017, 11:27pm UTC](https://community.openpbs.org/t/pp-482-non-destructive-walltime/587/10 "2017-06-29T23:27:39Z")
OK I’ve updated the EDD. I’ve replaced Interface 2. The old extension factor is now gone. The new interface 2 is the error message returned when soft\_walltime is used with STF jobs. I’ve removed all mention of the extension factor. Now if a job exceeds its soft walltime, it will be extended by 100% of its soft\_walltime up until its hard walltime (if any). I’ve also done a bit of housekeeping. I have italicized all resource names (e.g., soft\_walltime). I’ve moved the word hard to the front of the newly italicized walltime to read a bit more smoothly. Bhroam ---
**Author:** ![jon](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/jon/32/23_2.png) [@jon](https://community.openpbs.org/u/jon)\ **Post date:** [July 7, 2017, 3:06pm UTC](https://community.openpbs.org/t/pp-482-non-destructive-walltime/587/11 "2017-07-07T15:06:02Z")
Looks good. I sign off. One thing you may consider changing is to define STF before you use it in the details for interface one. ---
**Author:** ![billnitzberg](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/billnitzberg/32/6_2.png) [@billnitzberg](https://community.openpbs.org/u/billnitzberg)\ **Post date:** [July 7, 2017, 3:16pm UTC](https://community.openpbs.org/t/pp-482-non-destructive-walltime/587/12 "2017-07-07T15:16:20Z")
Re: The most recent design change of setting estimated.soft\_walltime Can you provide some background here – this addition came as a last minute surprise. What’s the goal, how will it be used, when is it set, what is it set to before soft\_walltime is exceeded, does it get recorded in the accounting log, etc. Thx! ---
**Author:** ![agurban](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/agurban/32/67_2.png) [@agurban](https://community.openpbs.org/u/agurban)\ **Post date:** [July 7, 2017, 4:33pm UTC](https://community.openpbs.org/t/pp-482-non-destructive-walltime/587/13 "2017-07-07T16:33:04Z")
Yes, please do add answers to Bill’s questions. Otherwise I’ll have to ask them again when it’s time to document the work. Thanks. ---
**Author:** ![iestockdale](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/iestockdale/32/72_2.png) [@iestockdale](https://community.openpbs.org/u/iestockdale)\ **Post date:** [July 7, 2017, 6:03pm UTC](https://community.openpbs.org/t/pp-482-non-destructive-walltime/587/14 "2017-07-07T18:03:22Z")
Bhroam - I would add, given the expression of surprise, could you mention what requirement and user story is being satisfied by this new interface. Does this satisfy some part of what’s in the description for PP-482 that was previously left uncovered? ---
**Author:** ![bhroam](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bhroam/32/20_2.png) [@bhroam](https://community.openpbs.org/u/bhroam)\ **Post date:** [July 7, 2017, 7:59pm UTC](https://community.openpbs.org/t/pp-482-non-destructive-walltime/587/15 "2017-07-07T19:59:19Z")
Hey, Knowing this is the second update to the feature, I’m not sure I’d call estimated.soft\_walltime a last minute surprise. Before I said that there was no way the user could find out the current soft walltime of their job. I’m adding an interface that allows them to see what the current soft walltime is set to. I think this would be useful. If a top job keeps being pushed out by a job that keeps being extended, admins can use estimated.soft\_walltime to figure out why. If you think the attribute isn’t useful, please let me know. As for the use case it satisfies, there is only one user story for the entire feature. All the interfaces fall under it. As for the accounting log, no, the soft walltime is not recorded. It’s really only useful while the job is running. The resources\_used.walltime is the information that is interesting to be printed in the accounting log. If someone wants to, they can do the math and figure out what the soft walltime was extended to after the job has run. ---
**Author:** ![iestockdale](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/iestockdale/32/72_2.png) [@iestockdale](https://community.openpbs.org/u/iestockdale)\ **Post date:** [July 8, 2017, 12:33am UTC](https://community.openpbs.org/t/pp-482-non-destructive-walltime/587/16 "2017-07-08T00:33:16Z")
Bhroam, in regard to understanding the motivation for estimated.soft\_walltime, it would be helpful to have something in the use case to which it could be tied. The closest thing right now is this sentence: “By introducing soft walltime admins will have the opportunity to look at usage statistics and predict times that should be much closer to reality then no walltime or very padded walltimes”, but I don’t think that does it. ---
**Author:** ![jon](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/jon/32/23_2.png) [@jon](https://community.openpbs.org/u/jon)\ **Post date:** [July 10, 2017, 4:15pm UTC](https://community.openpbs.org/t/pp-482-non-destructive-walltime/587/17 "2017-07-10T16:15:02Z")
> [@bhroam](#): > > If a top job keeps being pushed out by a job that keeps being extended, admins can use estimated.soft\_walltime to figure out why. If you think the attribute isn’t useful, please let me know. I’m not sure how useful estimated.soft\_walltime would be. I do think that a counter being incremented for each time the soft\_walltime was increased that was stored in the accounting logs would be useful. This way admin can see how many times the predicted soft walltime was incremented. This would allow the admin to easily do further analysis when trying to improve the prediction ---
**Author:** ![bhroam](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bhroam/32/20_2.png) [@bhroam](https://community.openpbs.org/u/bhroam)\ **Post date:** [July 19, 2017, 12:37am UTC](https://community.openpbs.org/t/pp-482-non-destructive-walltime/587/18 "2017-07-19T00:37:41Z")
I thought of something that hasn’t been discussed yet. What should qstat report? Right now qstat will either print cput or walltime. In the case of soft\_walltime, the walltime is not the duration we consider when scheduling. Should we still be printing walltime? Should we print soft\_walltime? If we print soft\_walltime, then what happens when we exceed the soft\_walltime estimate? Should we continue printing an old estimate that is no longer valid? Should we print the most up to date one? If so, that is a reason for estimated.soft\_walltime. I also realized that I was wrong about one thing. Just knowing how many times a job has exceeded its soft\_walltime doesn’t necessarily tell you what the final soft\_walltime estimate is. If a job’s soft\_walltime estimate would ever be extended past its walltime, it is set to the walltime. I don’t find printing the number of times the soft\_walltime has been exceeded all that useful in the accounting logs. The amount of time the job really took (resources\_used.walltime) is much more useful than the number of times a job’s soft\_walltime was extended. It’s telling you exactly how far off you were. That is in the accounting logs already. I made a small change to the EDD explaining what will happen if you submit a job with cput. The answer is what you’d expect, the job is killed when it reaches its cput limit. Bhroam ---
**Author:** ![scc](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/scc/32/234_2.png) [@scc](https://community.openpbs.org/u/scc)\ **Post date:** [July 21, 2017, 7:31pm UTC](https://community.openpbs.org/t/pp-482-non-destructive-walltime/587/19 "2017-07-21T19:31:49Z")
Just to make sure we are on the same page, you are talking about the “Req’d Time” column in the “alternate” qstat output format, right? If so my opinion is that leaving things as they are currently is correct and we do not show soft\_walltime in this field at all. That is, if CPU time is requested, “Req’d Time” shows CPU time. Otherwise, shows walltime. If neither were requested it should continue to show “–” (even if soft\_walltime was specified for the job). No job can ever “request” soft\_walltime, it can only be assigned to a job by a manager (usually through a hook). Showing a value to users that they cannot request in a column called “requested” would be confusing. It is true that today jobs are often “assigned” a walltime by either a hook or a resources\_default.walltime or resources\_max.walltime and these do show up in the “Req’d Time” field, but walltime can of course actually be requested by normal job submitters. Further, normal users will not normally even need to know about soft\_walltime since it is really a tool to help the scheduler make better decisions, not something that impacts/limits their jobs (especially after they are running) like hard walltime is. One question came into my mind while re-reading the EDD: Can a PBS manager can submit a job that explicitly requests soft\_walltime, or is it always read only outside of hooks? The current EDD makes me think the answer is that a manager can explicitly request it, but it would be nice to be sure. Also, what message would a user expect to see if they attempt to request it? “qsub: Cannot set attribute, read only or insufficient permission Resource\_List.soft\_walltime”? ---
**Author:** ![bhroam](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bhroam/32/20_2.png) [@bhroam](https://community.openpbs.org/u/bhroam)\ **Post date:** [July 21, 2017, 11:45pm UTC](https://community.openpbs.org/t/pp-482-non-destructive-walltime/587/20 "2017-07-21T23:45:22Z")
Thanks for your response Scott. You have made a good point about requested time. The user can directly set soft\_walltime if the site wants to set it up that way. I have an example queue job hook that allows it to be set. That being said, I agree that it’s not usually going to be set that way and we shouldn’t show it under a field named requested. You make another interesting point. That users don’t really need to know about soft\_walltime. It is a tool used by the scheduler, and doesn’t really affect the user. Does the user need to see soft\_walltime at all? Instead of making it only settable by a manager, should it be only manager read as well? The need or lack of need of estimated.soft\_walltime is still in question. I need a way to test soft\_walltime in my automated tests. I have two options. First is estimated.soft\_walltime. It can be easily queried by the automated test. If we decide to only have soft\_walltime be viewable by managers, estimated.soft\_walltime would only be viewable by managers as well. The other option is to print a scheduler log message every cycle for every job that has exceeded its soft\_walltime. I can bury this at DEBUG3, but it can still be pretty prolific every scheduling cycle. As for your question about managers submitting jobs with soft\_walltime, the answer is no. Every qsub has user perms. A manager can turn around and qalter the job, but they can’t submit a job with soft\_walltime. I made the requested updates to the EDD. I removed estimated.soft\_walltime for the time being, but it might have to go back for testing purposes. Bhroam [Next page](https://community.openpbs.org/t/pp-482-non-destructive-walltime/587.md?page=2)