Does PBS allow changing queue priorities dynamically?
At our site, we have large (highest priority), medium and small (lowest priority) queues and a highly oversubscribed load. The queue run limit is 1 for large.
Still, a large fraction of #jobs is in small queue. Is it possible to design a policy whereby if there are a SUFFICIENT # of waiting jobs in small queue, the top job from large queue is delayed to clear out small jobs en masse? We would like to do this without resorting to starvation.
Yes, you can change the queue priorities using qmgr command, this change is considered in the next scheduling cycle.
If all the jobs submitted to queues should have a walltime requested then you can have this policy set in the scheduler configuration
$PBS_HOME/sched_priv/sched_config: - kill -HUP after updating sched_config
help_starving_jobs: false ALL
strict_ordering: true ALL
backfill: true ALL
Server Configuration:
qmgr -c "set server backfill_depth=10"
Note: if walltime is not requested for a job, then the default walltime of a job is set to 5 years. Hence, please make sure default walltime is set at the server level if in case no jobs request walltime as below
qmgr -c “set server resources_default.walltime=01:00:00”