We have a set of gp100 nodes and a set of v100 gpu enabled nodes. We would like to set them up so that if a job requests a gpu_type = gp100 to get routed to the correct queues. We would like to do this without having to write a hook to push the nodes into the correct queue if possible.
pbs_version = 2020.1.0.20200720152739
create queue vis set queue vis queue_type = Execution set queue vis from_route_only = True set queue vis resources_max.gpu_type = gp100 set queue vis resources_max.ngpus = 1 set queue vis resources_min.gpu_type = gp100 set queue vis resources_min.ngpus = 1 set queue vis default_chunk.Qlist = vis set queue vis enabled = True set queue vis started = True create queue gpu set queue gpu queue_type = Execution set queue gpu from_route_only = True set queue gpu resources_max.gpu_type = v100 set queue gpu resources_min.gpu_type = v100 set queue gpu resources_min.ngpus = 1 set queue gpu default_chunk.Qlist = gpu set queue gpu enabled = True set queue gpu started = True Job resource request: #PBS -l select=1:ncpus=1:mem=10G:gpu_type=v100:ngpus=1
We would expect it to get pushed to the GPU queue, but this job instead gets pushed to vis.
We have this in our sched_config:
resources: "ncpus, mem, gpu_type, ngpus, arch, host, vnode, aoe, eoe, Qlist"
And resource define as below
create resource gpu_type set resource gpu_type type = string set resource gpu_type flag = h
Any guidance or advice from others that have solved this issue would be greatly appreciated.