I am fishing for hints/advises for pbs-pro configuration for a cluster used for “operational production” (high-prio jobs run a set number of times per day).
Our old cluster runs a different scheduler, and we want to move to PBS pro along a “path of least resistance”. Without bending PBS pro too much - but also without having to rewrite scripts, jobs guides etc more than we have to.
My present query is related to how reservations work in PBS pro - vs how we are “used to” reservations working.
We have an operational user (“prod”), which automatically runs all the high-prio jobs. Other users submit jobs for eg. testing, development and hindcasting.
We enforce a 1.5h maximum wall time on all jobs - and since we have full control of users, no jobs will ask to get the walltime extended during run time. An “operational reservation” can be submitted more than 2 hours before it is used. In principle, this does not actually have to be implemented as a “reservation”. Anything, which will keep other users jobs away from the “reserved” nodes will do. Maybe hooks is the way to go - I don’t know PBS pro well enough to make that decision. Preferably, jobs from user prod should be able to run on any nodes - even spanning to nodes, which are not explicitly part of the “reservation”. It does not really matter how much other jobs/users are delayed in that process.
Preferably, our “job control system”, which actually submits operational jobs to the queue/scheduler, should not need to know anything about reservation queues and similar things.
On the “old system” all jobs are submitted to a single queue, i.e. there are no “reservation queues”.
Our node usage is always job-exclusive: mom are set up with “sharing = force_exclhost”. To help the scheduler, jobs are also submitted with -l place=exclhost (typically even scatter:exclhost).
Thus, basically I am looking for ways to have PBS pro “reserve” some resources/nodes in time - without creating “boundaries” for the jobs in time or space (spanning nodes).
I cannot line up all the jobs prior to the start of the “operational compute cycle”, as some jobs will depend on various external parameters - also stuff, which is unknown at the onset of the compute cycle. Also, it is quite normal, that there may be a few minutes at a time, where the queue is seemingly empty of operational jobs, and it is important that the scheduler does not start low-priority jobs at these times. It is also normal, that some operational jobs may start “ahead of time” - as soon as they are ready to go, rather than waiting for the onset of the reservation.
If I use what seems to be the standard reservation in PBS pro (
pbs_rsub), then apparently I must submit (or move) jobs to the reservation queue in order to use the reservation. Also, jobs cannot start before the reservation onset - and continue to run while the reservation becomes active. Although this is a smaller issue for us, jobs also seemingly cannot span reserved and non-reserved hosts.
Are there any commonly used / well-known ways to reserve resources in a “transparent to the jobs” way? A good way to do this would help me a lot in porting to PBS pro.
Please let me know if I ask the wrong questions, or if the questions do not make sense.