Routing queues and queuejob event hooks

Developers
1

Hi all,
We have a default queue in our configuration that routes jobs to several other execution queues according to certain resources, mostly walltime.

The question: is it possible to have openpbs executing the queuejob hooks after the job has moved to the final execution queue?

Right now, the order I observe is: first execute the hooks, then route the job, so all my queuejob hooks see the same default routing queue as all the job’s queue.

Cheers,
Pepe

2

Could you please share your hook on this forum (if possible) ? and purspose/goal of this hook.

ROUTE_QUEUE=str(pbs.event().job.queue)
if ROUTE_QUEUE=“shortq”:
else:

3

Hi Adarsh,

I cannot share the hook… but I will explain better what I’m seeing. I have set the default queue to, let’s say, “queue1”… this one is a routing queue that sends to “queue2” or “queue3”… etc. When users submit a job without specifying the queue with the -q option, my hook will get an empty value in job.queue… if they specify -q queue1, the value will be indeed “queue1” and not any of the destinations.

In another cluster I’m working with (it has pbspro), the job.queue will indeed show “queue2” or “queue3”, etc.

It is like if for openpbs the order after submission is:

  • execute queuejob hooks
  • route
  • execute other hooks

but for pbspro is:

  • route
  • execute queuejob hooks
  • execute other hooks

Is this a know difference between openpbs and pbspro? can the order be changed in openpbs to follow that of pbspro? I really don’t want to code in a hook the logic of the routing queue. Imagine if I add another destination to the routing queue, I would have to modify the hook(s) as well!.

Cheers,
Pepe

4

I have noticed the reported behavior as well. And to complete the response mentioned above, the routing queue looks like this:

create queue entry
set queue entry queue_type = Route
set queue entry route_destinations = default
set queue entry route_destinations += queue1
set queue entry route_destinations += queue2
...

Considering a queuejob hook:

# queuejob hook
...
myqueue = pbs.event().job.queue

Now, as a user, if I submit a job qsub -l select=..., then myqueue will be unset. User can always explicitly specify the queue switch (e.g. -q queue1) and then it works but the question is when the queue is unspecified.

One could, in principle, set to the default queue:

...
if myqueue  is None:
    myqueue  = pbs.server().default_queue
    #  myqueue is 'default' at this point

In this scenario, myqueue takes the value of the default queue, and not one of the queues that is should route to (queue1, queue2).

As @pepe_grenoble mentioned, pbspro seems to have a slightly different execution sequence: the queue routing takes place before the execution of the queuejob hook. In that case, myqueue already comes with the queue that is was routed to. So you could do:

# queuejob hook
...
myqueue = j.pbs.event().job.queue
if str(myqueue) == 'queue1':
    pass
elif str(myqueue) == 'queue2':
    pass

With this hook I am trying to apply different rules based on the queue that the job gets assigned to but I haven’t been able to do so in openpbs.

5

What version of PBSPro do you notice that behaviour in?

At least for 2021.1.1 queuejob hooks are executed before any routing is done from a top level routing queue much like you observed with openpbs.

6

PBSPro 2021.1.0

Perhaps is this a new “feature” in 2021.1.1?