# Ability to delete jobs via preemption **URL:** \ **Category:** Developers\ **Created:** [April 6, 2019, 12:32am UTC](https://community.openpbs.org/t/ability-to-delete-jobs-via-preemption/1541 "2019-04-06T00:32:27Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![bhroam](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bhroam/32/20_2.png) [@bhroam](https://community.openpbs.org/u/bhroam)\ **Post date:** [April 6, 2019, 12:32am UTC](https://community.openpbs.org/t/ability-to-delete-jobs-via-preemption/1541/1 "2019-04-06T00:32:27Z")
Currently jobs can be preempted via suspension, checkpointing, or requeuing. This RFE will add the ability to preempt via deleting the job. The design is here: [https://pbspro.atlassian.net/wiki/spaces/PD/pages/1197735939/Preemption+via+deletion](https://pbspro.atlassian.net/wiki/spaces/PD/pages/1197735939/Preemption+via+deletion) Bhroam ---
**Author:** ![agrawalravi90](https://avatars.discourse-cdn.com/v4/letter/a/848f3c/32.png) [@agrawalravi90](https://community.openpbs.org/u/agrawalravi90)\ **Post date:** [April 10, 2019, 5:29pm UTC](https://community.openpbs.org/t/ability-to-delete-jobs-via-preemption/1541/2 "2019-04-10T17:29:13Z")
Thanks for proposing this change Bhroam. Just one comment: “This means the set of letters accepted will be ‘SCRD’.” Can you please mention what the default value will be? will it be SCRD ? Should it be something else? Since you mentioned that C and R could be overridden by the job submitter, maybe we should make it SDCR ? Just thinking out loud. “This means if D is connected to C or R, and a job is either -c n or -r n, the job will be considered for preemption” not sure I understood that, how can D be connected to C or R ? Will there be an accounting record for the job that gets deleted? Will the server logs mention that a delete request from scheduler was received for the jobs that get deleted via preemption? ---
**Author:** ![bhroam](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bhroam/32/20_2.png) [@bhroam](https://community.openpbs.org/u/bhroam)\ **Post date:** [April 11, 2019, 12:34am UTC](https://community.openpbs.org/t/ability-to-delete-jobs-via-preemption/1541/3 "2019-04-11T00:34:56Z")
> [@agrawalravi90](#): > > Can you please mention what the default value will be? will it be SCRD ? Should it be something else? Since you mentioned that C and R could be overridden by the job submitter, maybe we should make it SDCR ? Just thinking out loud. It would be dangerous to update the default preempt\_order. Preemption would be more destructive than it was before. Realistically, if a job isn’t -r n, R should always work. In any case, before I added the D to the default, I’d need buy in from the PMs. What do you think @scc? > [@agrawalravi90](#): > > not sure I understood that, how can D be connected to C or R ? It has to do with how the scheduler picks jobs to preempt. If the job was -c n or -r n and we were C or R, the scheduler would notice that and ignore the job. I was trying to say that if we were RD and a job is -rn, the scheduler wouldn’t ignore the job. Now that I think about, that’s kind of obvious. I’ll just remove the statement. > [@agrawalravi90](#): > > Will there be an accounting record for the job that gets deleted? Will the server logs mention that a delete request from scheduler was received for the jobs that get deleted via preemption? The accounting records will not change. For the most part, preemption works just like root doing a qsig, qhold, or qrerun. Now we’re including qdel in the mix. This means the normal D accounting record will be printed. Bhroam ---
**Author:** ![agrawalravi90](https://avatars.discourse-cdn.com/v4/letter/a/848f3c/32.png) [@agrawalravi90](https://community.openpbs.org/u/agrawalravi90)\ **Post date:** [April 11, 2019, 6:44am UTC](https://community.openpbs.org/t/ability-to-delete-jobs-via-preemption/1541/4 "2019-04-11T06:44:03Z")
> [@bhroam](#): > > his means the normal D accounting record will be printed. Would it add any value to differentiate a normal D record with the one that happened due to preemption? What will the value of ‘requestor’ be in the D record? ---
**Author:** ![bhroam](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bhroam/32/20_2.png) [@bhroam](https://community.openpbs.org/u/bhroam)\ **Post date:** [April 11, 2019, 10:42pm UTC](https://community.openpbs.org/t/ability-to-delete-jobs-via-preemption/1541/5 "2019-04-11T22:42:09Z")
That’s a very good question, @agrawalravi90. The delete request would be internally generated inside the server, so I’m assuming it’ll be the server. I won’t be able to tell until I actually code up a POC. ---
**Author:** ![arungrover](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/arungrover/32/40_2.png) [@arungrover](https://community.openpbs.org/u/arungrover)\ **Post date:** [April 12, 2019, 9:56pm UTC](https://community.openpbs.org/t/ability-to-delete-jobs-via-preemption/1541/6 "2019-04-12T21:56:47Z")
I like the design proposal. I have a question since the server will wait for the job to be finished before it replies back to scheduler about preemption, I assume it will not use ‘force’ option to delete the job, is it correct? I do have two comments on the discussion going on here - - I think we should not update the default preempt\_order to add ‘D’ to it. This is because of the same reason as @bhroam stated, the job preempted this way will be irrecoverable. It is better decided by admins what they want to do instead of adding ‘D’ as part of the default value. - About accounting records, I don’t think anything should change for accounting logs as long as the server does not use ‘Force’ option to delete the job. Currently, we log ‘D’ and ‘E’ record for the jobs that are running and then deleted using qdel command. I guess that will not change when we apply preemption with deletion. ---
**Author:** ![lisa-altair](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/lisa-altair/32/643_2.png) [@lisa-altair](https://community.openpbs.org/u/lisa-altair)\ **Post date:** [April 12, 2019, 11:35pm UTC](https://community.openpbs.org/t/ability-to-delete-jobs-via-preemption/1541/7 "2019-04-12T23:35:49Z")
> [@agrawalravi90](#): > > Will the server logs mention that a delete request from scheduler was received for the jobs that get deleted via preemption? I have the same question as @agrawalravi90, did I miss the answer? I also agree with @arungrover, we should not by default add ‘D’ to the default preempt\_order. ---
**Author:** ![bhroam](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bhroam/32/20_2.png) [@bhroam](https://community.openpbs.org/u/bhroam)\ **Post date:** [April 15, 2019, 11:34pm UTC](https://community.openpbs.org/t/ability-to-delete-jobs-via-preemption/1541/8 "2019-04-15T23:34:17Z")
There will definitely be a scheduler log message. The scheduler prints a message like "Job preempted by " where is suspension, checkpoint, or requeue. We’d add a fourth for deletion. As for a server log message, I’m not 100% sure. I wasn’t going to add anything specifically for this enhancement, but a delete request will be generated internally. I suspect there will be a log message that comes with that. ---
**Author:** ![bhroam](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bhroam/32/20_2.png) [@bhroam](https://community.openpbs.org/u/bhroam)\ **Post date:** [April 16, 2019, 12:45am UTC](https://community.openpbs.org/t/ability-to-delete-jobs-via-preemption/1541/9 "2019-04-16T00:45:43Z")
I have updated the design with more of the internal design. Please take a look. Bhroam ---
**Author:** ![scc](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/scc/32/234_2.png) [@scc](https://community.openpbs.org/u/scc)\ **Post date:** [April 16, 2019, 2:53pm UTC](https://community.openpbs.org/t/ability-to-delete-jobs-via-preemption/1541/10 "2019-04-16T14:53:58Z")
I agree that D should be left out of the default preempt\_order so people are not surprised with job loss where it did not happen previously (even if it is viewed as a bug by some that qsub -rn jobs are immune to preemption by requeue today and there is really nothing that can be done without this feature). Regarding the accounting log, what if we added a new special negative exit\_status to indicate a job was deleted by preemption? I don’t think there’d be any real loss of useful information from not seeing the true exit\_status resulting from the job being killed by a signal. This is likely not needed if the D record does indeed show root@server\_host as the requestor (which I imagine it will). Thoughts on this? ---
**Author:** ![bhroam](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bhroam/32/20_2.png) [@bhroam](https://community.openpbs.org/u/bhroam)\ **Post date:** [May 7, 2019, 1:19am UTC](https://community.openpbs.org/t/ability-to-delete-jobs-via-preemption/1541/11 "2019-05-07T01:19:04Z")
Just to wrap this up. I will leave the default preempt\_order alone. I think changing the exit status is not the right way to go about informing the admin that the job was preempted. I think changing the requestor on the ‘D’ record to something like ‘Scheduler’ is the right way to go about it. I will see if I can make this happen. I’m still in the process of working out the kinks of merging this functionality into the new preemption framework. Bhroam