Unable to set FIFO with Strict Ordering and Backfilling

Users/Site Administrators
1

Hi all,

I am from Slurm background and new to openpbs.

I want to configure my Head node with FIFO with Strict Ordering and Backfilling

I have configured following parameters in “/myhpc/pbs/sched_priv/sched_config” and restarted the PBS service. but it seems not working .Can anybody help what would be the testing criteria ?

$qstat --version
pbs_version = 22.05.11

strict_ordering: True ALL
round_robin: False ALL
job_sort_key: (commented out)
fair_share False ALL
backfill_depth: 1
job_sort_formula: (unset)

Also I am trying to set the job sort formula as below . Let me know how to implement this and list the jobs according to priority order whicha re in queued state ?

qmgr -c "unset server job_sort_formula = ‘5job_priority + 10queue_priority + ncpus * walltime + 20eligible_time +100fairshare_factor’ "

is there any similar utility that of sprio in slurm which lists the jobs in priority order ?

2

Hi,

How to set the job sort formula?

qmgr <hit return>
qmgr: set server job_sort_formula = "<formula>"
quit

To unset:
qmgr <hit return>
qmgr: unset server  job_sort_formula 
quit

The job_sort_formula supersedes all job soring polices (job sort key, fairshare) . The jobs will be run in the order based on the value or weight calculated by the formula. (higher considered first).

formula is : ncpus * walltime
job1 : 100
job2: : 300
job3 : 200
The expected job sort order to run the job is : job2, job3, job1

  • you can add backfill in this case, if in case strict walltime request are maintained.
  • by_queue : prime / non-prime is set to false in the $PBS_HOME/sched_config

You need increase the scheduler log level to scan / extract the priority of the respective jobs.for their job sort formulat values. (there was a command , pbs_stat --eval-formula in the unsupported folder of $PBS_EXEC, that used to print the job sort formula values of the job).

If in case you have not , please refer this documentation

4.9.21 Using a Formula for Computing Job Execution Priority

Thank you

3

Thank you very much for such detailed feedback!

Could you please help to suggest if it is advisible to set the following forameters togather. In document It is suggested not to to use strict ordering and backfilling with fairshare.(*Ref: PBS Professional 2022.1 Administrator’s Guide, Section 4.9.3 *)

strict_ordering: True ALL
round_robin: False ALL
fair_share True ALL
by_queue: True prime
by_queue: True non_prime
job_sort_key: “job_priority HIGH” all
node_sort_key: “sort_priority HIGH” ALL
provision_policy: “aggressive_provision”
resources: “ncpus, mem, arch, host, vnode, aoe, eoe”
smp_cluster_dist: pack
fair_share: true ALL
fairshare_usage_res: cput
fairshare_entity: euser
fairshare_decay_time: 24:00:00
fairshare_decay_factor: 0.5
preemptive_sched: true ALL

set server eligible_time_enable = True
job_sort_formula: ‘5job_priority + 10queue_priority + ncpus * walltime + 20eligible_time+ 50fairshare_factor’
sched job_sort_formula_threshold =< somevalue>
set queue parallel backfill_depth = 10

4

Fairshare policy sorts the jobs based on the historical usage of the cluster. Hence, the recommendation of not to use strict ordering and backfilling.

  • fairshare supersedes other policies
  • job sort formula supersedes fairshare policy
  • you can include fairshare values as a factor in the job sort fomula, which means you can enable fairshare policy and job sort formula and include fairshare usage in your formula and decide the ordering of jobs as per the formula values Note: job sort formula is in play here and you can use backfill/strict ordering ( this gives us an opportunity to enable fairshare and use it as factor in job sort formula)

Seems like you have implemented the same here .

5

Thank You very much for the clarification.

1 Like