# How to route to multiple execution queues? **URL:** \ **Category:** Users/Site Administrators\ **Created:** [June 14, 2017, 12:16am UTC](https://community.openpbs.org/t/how-to-route-to-multiple-execution-queues/576 "2017-06-14T00:16:15Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Jeff](https://avatars.discourse-cdn.com/v4/letter/j/e19b73/32.png) [@Jeff](https://community.openpbs.org/u/Jeff)\ **Post date:** [June 14, 2017, 12:16am UTC](https://community.openpbs.org/t/how-to-route-to-multiple-execution-queues/576/1 "2017-06-14T00:16:15Z")
Hi, As this cluster has evolved, we are using a mix of hardware and operating systems on the compute nodes. These are split into separate execution queues. What I am missing is how to properly route to one and overflow to the other. Using the max\_queued limit gets close to a solution “max\_queued\_res.ncpus = [o:PBS\_ALL=260]” except for this counts held jobs as if they were running. The queued jobs in the routing queue that could be running are stuck waiting for held jobs to finish. I would expect that someone has a clean solution to this configuration. If this requires a queuejob hook, that’s fine. (It would be even better if someone has said hook available.) The Big Book shows nice ways to split out the queues by core count, or walltime, or memory size. What I did not find was how to feed one queue and when full, move on to another queue. The PBS Pro version is 13.0.2.153173. Many thanks, -Jeff ---
**Author:** ![dw808303](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/dw808303/32/15_2.png) [@dw808303](https://community.openpbs.org/u/dw808303)\ **Post date:** [June 14, 2017, 9:39pm UTC](https://community.openpbs.org/t/how-to-route-to-multiple-execution-queues/576/2 "2017-06-14T21:39:37Z")
Hello Jeff, Could you paste your route queue and destination queue definitions from qmgr here? Feel free to anonymize the data (e.g. machine names, etc.) if necessary. ---
**Author:** ![Jeff](https://avatars.discourse-cdn.com/v4/letter/j/e19b73/32.png) [@Jeff](https://community.openpbs.org/u/Jeff)\ **Post date:** [June 14, 2017, 11:28pm UTC](https://community.openpbs.org/t/how-to-route-to-multiple-execution-queues/576/3 "2017-06-14T23:28:01Z")
Hi Dave, Here’s most of the queue config info: [root@\ ~]# qstat -Qf Queue: gen8r65 queue\_type = Execution total\_jobs = 2 state\_count = Transit:0 Queued:0 Held:1 Waiting:0 Running:1 Exiting:0 Begun:0 max\_queued\_res.ncpus = [o:PBS\_ALL=40] resources\_assigned.mpiprocs = 20 resources\_assigned.ncpus = 20 resources\_assigned.nodect = 1 hasnodes = True enabled = True started = True Queue: gen9r65 queue\_type = Execution total\_jobs = 19 state\_count = Transit:0 Queued:0 Held:12 Waiting:0 Running:7 Exiting:0 Begun:0 max\_queued\_res.ncpus = [o:PBS\_ALL=180] resources\_assigned.mem = 3gb resources\_assigned.mpiprocs = 74 resources\_assigned.ncpus = 77 resources\_assigned.nodect = 7 hasnodes = True enabled = True started = True Queue: gen8r64 queue\_type = Execution total\_jobs = 26 state\_count = Transit:0 Queued:0 Held:3 Waiting:0 Running:23 Exiting:0 Begun:0 max\_queued\_res.ncpus = [o:PBS\_ALL=260] resources\_assigned.mem = 11gb resources\_assigned.mpiprocs = 220 resources\_assigned.ncpus = 241 resources\_assigned.nodect = 32 hasnodes = True enabled = True started = True Queue: ANSYS160 queue\_type = Route total\_jobs = 1 state\_count = Transit:0 Queued:0 Held:1 Waiting:0 Running:0 Exiting:0 Begun:0 route\_destinations = gen8r64,gen8r65,gen9r65 enabled = True started = True Queue: ANSYS171 queue\_type = Route total\_jobs = 0 state\_count = Transit:0 Queued:0 Held:0 Waiting:0 Running:0 Exiting:0 Begun:0 route\_destinations = gen9r65,gen8r65 enabled = True started = True Queue: app\_N queue\_type = Route total\_jobs = 0 state\_count = Transit:0 Queued:0 Held:0 Waiting:0 Running:0 Exiting:0 Begun:0 route\_destinations = gen8r64,gen9r65 enabled = True started = True Queue: app\_V queue\_type = Route total\_jobs = 0 state\_count = Transit:0 Queued:0 Held:0 Waiting:0 Running:0 Exiting:0 Begun:0 route\_destinations = gen8r64 enabled = True started = True Queue: default queue\_type = Route total\_jobs = 0 state\_count = Transit:0 Queued:0 Held:0 Waiting:0 Running:0 Exiting:0 Begun:0 route\_destinations = gen8r64 enabled = True started = True Queue: gen9 queue\_type = Route total\_jobs = 0 state\_count = Transit:0 Queued:0 Held:0 Waiting:0 Running:0 Exiting:0 Begun:0 route\_destinations = gen9r65 enabled = True started = True ---
**Author:** ![bhroam](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bhroam/32/20_2.png) [@bhroam](https://community.openpbs.org/u/bhroam)\ **Post date:** [June 15, 2017, 8:23pm UTC](https://community.openpbs.org/t/how-to-route-to-multiple-execution-queues/576/4 "2017-06-15T20:23:55Z")
I think the limit you want is max\_queued\_threshold instead of max\_queued. The max\_queued\_threshold limit deals with jobs in the queued state only whereas max\_queued is the total number of jobs. There is both a max\_queued\_threshold and a max\_queued\_threshold\_res. Bhroam ---
**Author:** ![scc](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/scc/32/234_2.png) [@scc](https://community.openpbs.org/u/scc)\ **Post date:** [June 15, 2017, 9:58pm UTC](https://community.openpbs.org/t/how-to-route-to-multiple-execution-queues/576/5 "2017-06-15T21:58:23Z")
Hi Bhroam, I think you are thinking of queued\_jobs\_threshold, right? If so, it does count held jobs: ``` [root@centos7 tmp]# qmgr -c "p q r" # # Create queues and set their attributes. # # # Create and define queue r # create queue r set queue r queue_type = Route set queue r route_destinations = workq set queue r route_destinations += workq2 set queue r enabled = True set queue r started = True [root@centos7 tmp]# qmgr -c "p q workq" # # Create queues and set their attributes. # # # Create and define queue workq # create queue workq set queue workq queue_type = Execution set queue workq from_route_only = True set queue workq kill_delay = 20 set queue workq enabled = True set queue workq started = True set queue workq queued_jobs_threshold = [o:PBS_ALL=4] [root@centos7 tmp]# qmgr -c "p q workq2" # # Create queues and set their attributes. # # # Create and define queue workq2 # create queue workq2 set queue workq2 queue_type = Execution set queue workq2 from_route_only = True set queue workq2 enabled = True set queue workq2 started = True [user1@centos7 ~]$ echo "sleep 1000000" | qsub 7303.centos7 [user1@centos7 ~]$ qhold 7303.centos7 [user1@centos7 ~]$ echo "sleep 1000000" | qsub 7304.centos7 [user1@centos7 ~]$ echo "sleep 1000000" | qsub 7305.centos7 [user1@centos7 ~]$ echo "sleep 1000000" | qsub 7306.centos7 [user1@centos7 ~]$ echo "sleep 1000000" | qsub 7307.centos7 [user1@centos7 ~]$ qstat Job id Name User Time Use S Queue ---------------- ---------------- ---------------- -------- - ----- 7303.centos7 STDIN user1 0 H workq 7304.centos7 STDIN user1 0 Q workq 7305.centos7 STDIN user1 0 Q workq 7306.centos7 STDIN user1 0 Q workq 7307.centos7 STDIN user1 0 Q workq2 ``` ---
**Author:** ![Jeff](https://avatars.discourse-cdn.com/v4/letter/j/e19b73/32.png) [@Jeff](https://community.openpbs.org/u/Jeff)\ **Post date:** [June 16, 2017, 6:08pm UTC](https://community.openpbs.org/t/how-to-route-to-multiple-execution-queues/576/6 "2017-06-16T18:08:52Z")
I tried using the suggested parameter of “queued\_jobs\_threshold\_res.ncpus = [o:PBS\_ALL=8]” and found that held jobs were still getting in the way of other jobs that could and should run. ``` test_head: Req'd Req'd Elap Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time ------------------------------ --------------- --------------- --------------- -------- ---- ----- ------ ----- - ----- 2373.test_head some_user ANSYS171 ANSYS -- 1 4 -- -- Q -- -- 2374.test_head some_user ANSYS171 ANSYS -- 1 4 -- -- Q -- -- 2375.test_head some_user ANSYS171 ANSYS -- 1 4 -- -- Q -- -- 2371.test_head some_user gen9r65 ANSYS -- 1 4 -- -- H -- -- 2372.test_head some_user gen9r65 ANSYS -- 1 4 -- -- H -- -- ``` It sounded like a good idea and was certainly worth a try. The queued jobs here seem to be blocked by the held jobs. The demonstration by scc looks like “queued\_jobs\_threshold” has the same problem with held jobs. This overall problem strikes me as if I’m missing something fundamental on how to properly configure the routing, or more precisely, how to configure the execution queues so that routing behaves as desired. I THOUGHT I had a good handle on how this works but that is clearly not the case. Thanks! ---
**Author:** ![scc](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/scc/32/234_2.png) [@scc](https://community.openpbs.org/u/scc)\ **Post date:** [June 16, 2017, 7:40pm UTC](https://community.openpbs.org/t/how-to-route-to-multiple-execution-queues/576/7 "2017-06-16T19:40:38Z")
Can you expand a bit on what your end goal is? There is something we call “internal peering” that might help you, depending on what exactly you are trying to achieve by having these jobs occupy different queues. In the “internal peering” scenario you could set up the queues to only take jobs from the routing (or execution, really) queue when they are able to run. ---
**Author:** ![Jeff](https://avatars.discourse-cdn.com/v4/letter/j/e19b73/32.png) [@Jeff](https://community.openpbs.org/u/Jeff)\ **Post date:** [June 16, 2017, 8:12pm UTC](https://community.openpbs.org/t/how-to-route-to-multiple-execution-queues/576/8 "2017-06-16T20:12:41Z")
Certainly! This use case here involves the following: -Hardware- There is a mix of two different hardware families (DL380 gen8 & gen9) and two different operating systems (RHEL6.4 & RHEL6.5), soon to be followed by RHEL7.3. The resulting permutations are mapped to execution queues for gen8r64, gen8r65, gen9r65, and there will be a gen9r73. -Software- The applications are a mix of large parallel ANSYS jobs and a large quantity of single thread (single core) jobs. We have evolving versions of ANSYS. The applications are only allowed to run on certified platforms. Each application and version has a routing queue that feeds only to the approved execution destinations. In the case of single destinations, the outcome is easy. In the case of multiple destinations, there is a preferred destination followed by a acceptable alternate destination. That is the situation that drives my original question of how to route to a primary execution queue while spilling over to an alternate execution queue if resources are available. The large parallel jobs and the small singular jobs need to run in a shared environment and play nice. CPU (or core) allocation seems like the right way to spread the mix of jobs in the execution queues and that was my original intent with this arrangement. The built-in configuration parameters seem to get really close to a working solution except for the held jobs getting in the way. Simple isolation of destinations would work fine except for the folks who would then have a problem with non-shared access, “Hey, how come my jobs are queued when there is room to run on those other nodes?” Thanks! ---
**Author:** ![Jeff](https://avatars.discourse-cdn.com/v4/letter/j/e19b73/32.png) [@Jeff](https://community.openpbs.org/u/Jeff)\ **Post date:** [August 7, 2017, 3:31pm UTC](https://community.openpbs.org/t/how-to-route-to-multiple-execution-queues/576/9 "2017-08-07T15:31:54Z")
This question remains unresolved. I needed to get the system back to a stable, unattended mode so I configured routing to single execution queues with no limits. It works but it also misses the idea of packing the available execution queues. If someone has a fix for this idea, please let me know. Thanks,