Clarification on backfill_depth and job execution order

Users/Site Administrators
1

Hello All,

In my test environment defult walltime is not mentioned anywhere in Queue /Job/server level. As a result Smaller jobs keep on fitting in between larger jobs and they are being blocked indefinitely by a steady stream of smaller, low-priority jobs until the weekend when PBS complex really becomes Idle.

Requirement : Lower Priority smaller jobs should only Go to Run Q after the scheduler has ensured enough resources has been reserved for top 3 pending Jobs in Priority Q as per job sort formula evaluation. No Job Preemption should happen

If I modify the settings as below and set following server parameters and enforce wall time in Jobs will it address above requirement ?

strict_ordering: True ALL
preemptive_sched: False ALL
job_sort_key: “job_priority HIGH” all
job_sort_formula = ‘ ncpus * walltime + eligible_time’

backfill_depth = 3 for Q name TestQ1.

==> Qmgr:set queue TestQ1 backfill_depth = 3

My understanding is as below. Can anyone confirm if that is correct?

PBS sets aside resources for the top 3 jobs in strict priority order before examining other jobs. That is, PBS looks at the currently running jobs to see when they will finish (using the wall-time estimates). From those finish times, PBS decides when enough resources (such as CPUs, memory, ,dyn resources etc…) will become available to run the top job.

PBS then creates a virtual reservation for those resources at that time. Now, when PBS looks at other jobs to see if they can start immediately, it also checks whether starting the job would collide with one of these reservations. Only if there are no collisions will PBS start the lower priority jobs.

2

If the jobs do not have walltime in their request then it is difficult for the scheduler to estimate the start time of the jobs . The backfill is done to support jobs that do not affect the start time of other jobs which are calendered to run based on the strict ordering.

If jobs are submitted without walltime request, then the default_walltime is 5 years. Any jobs which does not have the walltime, would affect the estimations.

This becomes obsolete if you use job_sort_formula. Job sort formula supercedes all sorting policies and fairshare policy.

  • If you are 100% sure you will have constant stream of large number of small jobs ( requiring less walltime), then create a separate queue for these jobs. However, this is based that these jobs does not request walltime and assumed they are smaller jobs requiring walltime.
  • you can write a hook or use queue defaults to set walltime for jobs that are not requesting walltime. Or even reject the jobs that does not ask for walltime.

Your understanding is correct, the backfill depth is a method of increasing the top jobs, that are cosidered to run next based on the situation of your cluster.

3

Thank you very much for quick response.

1 Like