I am configuring PbsPro (v14.1.0) to use two kinds of nodes, labeled “compute” and “io”. Basically, “compute” are reasonably new HPC nodes, while “io” are older.
The idea is that:
A. No job should span the two groups (each job must fit within one of the groups).
B. A job may explicitly ask for compute-nodes and get those
(In principle a job may also ask for io-nodes, but that is not critical).
C. Jobs not asking for compute-nodes should run on io-nodes, if any are available (this could be a priority thing).
D. If (for some reason) no io-nodes are available, then all jobs must be able to run on the compute nodes.
Here is what I find so far (on the present test system #compute > #io)
I have tried to use server settings node_group_enable and node_group_key to divide the nodes into two groups. That seems to work, but
a. Jobs tend to run on the smallest of the two sets (happens to be the io nodes on the present setup, but I would like to be able to control it). This feature seems documented, but I am not sure how to get around it).
b. If I ask for more nodes than what is presently in “compute”, then I will get all the compute nodes plus extra nodes from the io-group. I would rather have such a job request hang or even fail rather than spanning the two groups.
I have tried configuring the above by defining a new resource (as string or string_array), and then setting the resource_available for each node. Example:
qmgr -c "create resource nodetype type=string, flag=h"
vim /var/spool/pbs/sched_priv/sched_config - allow scheduling to be based on nodetype
qmgr -c “set node dn008 resources_available.nodetype=“io”” # For all io-nodes
qmgr -c “set node dn101 resources_available.nodetype=“compute”” # For all compute-nodes
As an alternative, I have tried to define resources “io” and “compute” (as type long or boolean) and define the node as eg.:
qmgr -c "set node NOD resources_available.io = True"
qmgr -c “set node NOD resources_available.compute = False”
However, if I then specify to get, say, one compute-node:
qsub -l compute=true …
then I get a reply (qstat job comment) noting:
comment = Can Never Run: Insufficient amount of queue resource: compute (True != False)
Similar if I try -l io=true.
However, if I use eg. “-l compute=false” then the job starts on an io-node. So what’s up with that?
Even with this “negated” setting, asking for more nodes that what is present in a single set, then I get allocated extra nodes from “the other set”, which is not what I want.
Presently, I have a setup with 5 compute nodes, so if I ask for six - like this:
qsub -I -l io=false,nodes=6
Then I get the 5 compute nodes plus one io-node.
I imagine that this is a “pretty common” setup to try to make.
Does anybody have idea/links on what to try?
I have already browsed the admin and reference guides (not read cover-to-cover, though).
And I have tried to follow this recipe:
without much success.
Eventually, I might want to try to let the queue choise select the node group, but for now I just want to make it work from qsub.
All kinds of hints, links and help will be much appreciated.