Hello All,
In our HPC environment with 80 nodes and we’ve 3 queues with different priority.
Now I want to implement the following mechanism when all 80 nodes are fully occupied :
• When a job comes in the high priority queues it put on hold running jobs of the lowest priority queue.
• Whenever there is a free resource then we resume the jobs from low priority queue and continue their execution.
Usually, high priority queues are associated with strict waltime limits. The high priority queues are sometimes used to run quick check runs / quick results . However, some might have these demands or priority work all the time, then they go for reservations, at least there is a guarantee .
• When a job comes in the high priority queues it put on hold running jobs of the lowest priority queue.
You can use preemption to suspend the jobs from the low priority queue. However, when the resources are availalbe , the low priority jobs that are suspended on the compute nodes cannot be moved/migrated to the now available free compute nodes.
if i understand in a different way, until there are no jobs are there in the high priority queue (provided the jobs ask for the resources that can be satisfied by the active/live cluster) , keep the low priority queue disabled, once high priority queue is empty , enable the low prirority queue ( keep the preemption disable in this case.