Routing GPU jobs to specific queues based on gpu model


We have a set of gp100 nodes and a set of v100 gpu enabled nodes. We would like to set them up so that if a job requests a gpu_type = gp100 to get routed to the correct queues. We would like to do this without having to write a hook to push the nodes into the correct queue if possible.

pbs_version = 2020.1.0.20200720152739

create queue vis
set queue vis queue_type = Execution
set queue vis from_route_only = True
set queue vis resources_max.gpu_type = gp100
set queue vis resources_max.ngpus = 1
set queue vis resources_min.gpu_type = gp100
set queue vis resources_min.ngpus = 1
set queue vis default_chunk.Qlist = vis
set queue vis enabled = True
set queue vis started = True

create queue gpu
set queue gpu queue_type = Execution
set queue gpu from_route_only = True
set queue gpu resources_max.gpu_type = v100
set queue gpu resources_min.gpu_type = v100
set queue gpu resources_min.ngpus = 1
set queue gpu default_chunk.Qlist = gpu
set queue gpu enabled = True
set queue gpu started = True

Job resource request:
#PBS -l select=1:ncpus=1:mem=10G:gpu_type=v100:ngpus=1

We would expect it to get pushed to the GPU queue, but this job instead gets pushed to vis.

We have this in our sched_config:
resources: "ncpus, mem, gpu_type, ngpus, arch, host, vnode, aoe, eoe, Qlist"

And resource define as below

create resource gpu_type
set resource gpu_type type = string
set resource gpu_type flag = h

Any guidance or advice from others that have solved this issue would be greatly appreciated.


Could you please unset the below and try again

qmgr : u s queue vis resources_max.ngpus
qmgr : u s queue vis resources_min.ngpus
qmgr : u s gpu resources_min.ngpus
qmgr : u s gpu resource_max.ngpus

Please share your pbsnodes -av output, if un-setting the above did not work .