How to control PBS sched to run GPU or cpu jobs

Users/Site Administrators
1

PBS queue has been implemented with FIFO method with fairshare. However the challenge is when CPU node( has only cpus) is free but PBS sends job to the GPU node( has cpu and gpu). How to tell PBS to first try all CPU node if not free then go to GPU node. If its GPU jobs then it should immediately start at GPU.

Any help with the configuration?

2

Soln 1:
During addition of compute nodes to the PBS Server:

  1. add the cpu nodes first - qmgr -c “create node cpu-node(s)”
  2. add the gpu nodes at the last - qmgr -c “create node gpu-node(s)”
  3. The pbsnodes -av | grep ^[a-zA-Z] output shows this order
    This will make sure the cpu nodes are checked first before moving into gpu nodes

Soln 2:
1 set cpu nodes priority value to 100
2. set gpu nodes priority value to 200
3. In the $PBS_HOME/sched_priv/sched_config , update the below node_sort_key line and kill -HUP
node_sort_key: “sort_priority LOW” ALL
4. This will make sure nodes are sorted from low to high

Hope this works out well for you

4

Hi,

soln1.

Below node order is already in place. however still gpu node has always taken first over free cpu node

pbsnodes -av | grep ^[a-zA-Z]
node10
node11
node4
node5
node12
node7
gpu01

soln2.

sched_config as node sort HIGH all. So is it possible to give cpus 200 and gpus 100 priority. if yes how to do it in below options.

Below is qmgr config of nodes.

Create and define node gpu01

create node gpu01
set node gpu01 state = free
set node gpu01 resources_available.arch = linux
set node gpu01 resources_available.host = gpu01
set node gpu01 resources_available.mem = 97225652kb
set node gpu01 resources_available.ncpus = 32
set node gpu01 resources_available.ngpus = 1
set node gpu01 resources_available.vnode = gpu01
set node gpu01 resv_enable = True
Qmgr: print node node1

Create nodes and set their properties.

Create and define node node1

create node node1
set node node1 state = job-busy
set node node1 resources_available.arch = linux
set node node1 resources_available.host = node1
set node node1 resources_available.mem = 97229404kb
set node node1 resources_available.ncpus = 32
set node node1 resources_available.vnode = node1
set node node1 resv_enable = True
Qmgr:

–

Regards,

Anilkumar Naik

5

Please set these priorities by running the below commands:

for i in node10 node11 node4 node5 node12 node7;do qmgr -c “set node $i priority=1000”; done

qmgr -c “set node gpu01 prirority=100”

In the $PBS_HOME/sched_priv/sched_config , update the below node_sort_key line and kill -HUP
node_sort_key: “sort_priority HIGH” ALL

[ kill -HUP < PID of pbs scheduler > to reread any updates made to sched_config ]

6

Dear Adarsh,

Thanks for the steps. it helped to implement node sorting.

node_sort_key: “sort_priority HIGH” ALL was already set in the sched_config. So do we need to still restart pbs sched? Or it will be automatically now sort the nodes as per the latest priority set to each nodes.

1 Like