we are currently facing a very weird behavior. Our system uses multiple “general” queues (for short/normal/long jobs, etc.). These queues have the same priority . Beside that, we have also several “high priority” queues (their priority is higher than for the short, normal and long queues) and job_sort_formula = queue_priority. Moreover, jobs are ordered by fair-share , and by_queue = false, i.e., we do not sort jobs in a queue-by-queue fashion but “all at once” (for the queues with the same priority).
Also, we have backfilling and strict_ordering enabled. Backfill_depth is set separately for each queue (depth = 10) and the opt_backfill_fuzzy = high. The expected behavior is that the scheduler first considers those high priority queues, and next all jobs from those short/normal/long queues are considered together (ordered by fair-share). When backfilling starts to schedule jobs, it should give a reservation to every top job that cannot start immediately. Now to the problem:
We have observed a weird behavior, where a high priority job (“high_job” of a user “high_user”) obtained a reservation since it was a top job. However, this reservation was repeatedly postponed to a later time by a different lower priority job (“low_job” of a user “low_user”). According to the log, both jobs were top jobs and according to the fair-share, “high_job” had a higher priority. We’ve checked this by running the pbsfs -c command (pbsfs -c high_user low_user --> high_user).
In our understanding, if a high priority top job receives a reservation it should not be postponed by a lower priority top job because the lower priority top job is selected later in the backfill process, right? We expect the lower priority top job to “see” that there is already a reservation for another (high priority) top job, thus seeking another (later) time slot. Instead, in our case, the latter lower priority top job simply ignored the first reservation and postponed the first (high priority) top job. We are unable to identify the reason for this behavior.
Could the problem be somehow related to the fact that those two jobs were from different queues (yet having the same queue priority)? Since we have backfill_depth set per queue and by_queue=false, once all jobs are ordered by fair-share those “top jobs” can be actually quite scattered within that sorted job list (as jobs come from different queues).
The problem is quite painful, since the high priority job is being postponed repeatedly for several days now and the user is complaining…
(Note: we are aware that backfill+fair-share is not a recommended combination, yet the observed behavior is somehow inexplicable for us.)