We are trying out fairshare with queue priority, and I’m wondering how “large” jobs will be affected by this policy. The Admin Guide says,
At each scheduling cycle, the scheduler attempts to run as many jobs as possible. It selects the most deserving job, runs it if it can, then recalculates to find the next most deserving job, runs it if it can, and so on.
I’m concerned with the situation where the most-deserving entity submits a “large” job on a heavily-utilized cluster, followed by less-deserving entities submitting large numbers of “small” jobs. Based on the description above, it seems that the scheduler would try to schedule the large job first, see that there aren’t enough resources available, then proceed to schedule the small jobs instead. If small jobs continue to be submitted, I would think that the large job would be starved for resources and never schedule.
Is this a concern with fairshare policy for the situation I described, or does the scheduler have some way of ensuring that less-deserving jobs get held until the large job is able to run? It sounds like backfilling and help_starving_jobs are not options with fairshare.
You can use top jobs and starving jobs with fairshare. They just don’t play very well together.
The situation you can get into is where you have a top job. You one of the most deserving entities. We add your top job to the calendar and estimate where and when it will run. We won’t use those resources for other jobs. What can happen is your other running jobs are still accumulating usage. If your usage grows high enough, you are no longer one of the most deserving entities. Your job loses its top job slot. Once the usage balances out again, your job will need to wait for those resources all over again.
Now we do our best to avoid this situation. When a top job is added to the calendar, we temporarily add the job’s requested resources to its fairshare usage. This will lower the entity’s priority during that cycle. Now that the entity has a lower priority, their jobs are less likely to run. Less likely doesn’t mean the situation is completely avoided.
From your use cases above, it does sound that top jobs and fairshare are what you want. Just keep in mind that this unfortunate situation can occur.
Yes, setting strict_ordering will turn on top job support. The number of top jobs at one time can be controlled by the server or queue backfill_depth attribute. By default you get one top job.