[Preempt queue] Jobs in queue1 should stop to let jobs in queue2 start

Users/Site Administrators
1

Hello,

Assuming:

  • I have two queues: low_priority and high_priority
  • I use a server_dyn_res which decide whether or not my job could run based on application license availabilities

Say I have 50 application licenses and 45 are currently in use by jobs in the “low_priority” queue.
I want to deploy a new job on “high_priority” queue which will require 10 licenses.

At job submission, my new job will be queued due to lack of licenses. At this time, I’d like to suspend one job from “low_priority” queue (ideally the one which was started last) to automatically free up some ressources for the job in my “high_priority” queue.
I tried to create some walltime/priorities settings without success. I also looked at the preempt queue http://docs.adaptivecomputing.com/mwm/archive/6-0/8.4preemption.php but I’m not sure what I’m supposed to do with that (eg: which file to configure to enable this?)

Any idea how to achieve this ?

2

Queue based preemption in PBS is based on the queue’s priority attribute. By default, any queue that has a priority >= 150 is considered an express queue. Jobs in an express queue can preempt jobs from normal priority queues. This is on by default.

That being said, preemption and server_dyn_res usually don’t play well together. Let me explain. Usually the way server_dyn_res works is you return the number of free licenses. The scheduler determines if your job can run if the number of licenses requested <= resources_available.licenses - resources_assigned.licenses. The way server_dyn_res works is to replace the resources_available value and to not use resources_assigned. This means you are placing a cap on the number of licenses the scheduler knows about for the entire cycle. So if your server_dyn_res script returns 5, then there is a max cap of 5 for the cycle. No matter how many jobs are preempted will increase that cap.

There is a way to make them place nice, but it requires more work.
The gist of it goes back to that formula the scheduler uses to determine if your job can run. Since server_dyn_res sets the resources_available value, you need to set this value to the number free + number allocated by PBS. This basically is the total number of licenses - number used outside of PBS. The next step is to tell PBS to keep track of how many licenses it has allocated.

An example might make things more clear:
Number of total licenses available: 40
Number of licenses used by PBS: 30
Number of licenses used outside of PBS: 5
Number of free licenses: 5

You need to have the server_dyn_res to return 35. This is the number that is used by pbs + the number free.

So let’s go back to the formula above:
requested <= resources_available - resources_assigned.

You need to tell PBS to keep track of the number of licenses allocated. You do this by setting the ‘q’ flag on your license resource.

Putting this all together:
create resource license type=long flag=q
in your server_dyn_res script, run qstat -Bf and find resources_available.licenses.
Add this to the number of free licenses that lmstat returns.
Have the server_dyn_res script return this value.

With all that said, if all the licenses are being used by PBS only, it makes this an easier problem to solve. Since the number free + number used by PBS = total number of licenses, you just get the total number of licenses from lmstat and return that.

As a note, adaptive handles preemption differently than PBS. I’m not sure how helpful their documentation will be.

Bhroam