Is it possible to submit a job to one of the nodes from a list of nodes?

Users/Site Administrators
1

Hello,

In PBS Pro, is it possible to submit a job to one of the nodes from a list of nodes?

example,
user1 wants to submit a job to one of the nodes from the list of nodes node1, node2, node3

These nodes are part of the same PBS complex, and this complex has more nodes than node1, node2, node3. But user1 at a given moment just wants to consider node1, node2, node3, and wants to submit a job to one of these three nodes.

Thanks,

2

Yes, there are many ways to achieve this, some of them are as below.

  • If the user wants to submit to only one specifc node :

qsub -l select=1:ncpus=1:mem=100mb:host=node1 – /bin/sleep 100

  • If the user wants to submit to more than one node:
    qsub -l nodes=node1+node2+node3 – /bin/sleep 100

Also, the same can be achieved by tagging the nodes with custom resources of user1 / userX choice and request those host level custom resources when requesting the job, in this case scheduler will appropriately schedule the job and user need not have to specify the hostname of the compute node explicitly.

1 Like
3

Thank you for the response.

The use case of submitting to exactly one node is clear. The question how to submit to any one node from a given list of nodes.

During my testing what I have observed is that the command, qsub -l nodes=node1+node2+node3 – /bin/sleep 100 , tries to check if all the nodes are available, if one of the nodes is not available for some reason, but the other two are available, the job goes in a pending state, it does not get submitted to one of the other two available hosts.

Thanks,

4

It’s possible for an administrator to create a custom resource and assign values accordingly. For example, if I create a custom resource called “color” and assign some nodes “red”, “blue”, or “green” the scheduler can be made to use this by adding “color” to the resources line in sched_config. This isn’t possible for an unprivileged user, but depending on your specific use case it may work for you.

Thanks,

Mike

5

Hello,

Can you describe your use case for needing to choose from specific nodes? Perhaps we can work out a good solution if we know what you need. I ask because PBS is designed to get you the resources you need without you having to know where those resources come from. You should be able to simply specify what you need, for example, N cpus, M memory, X software, and H hosts. Thanks.

6

++ Mike + Anne

Please follow the below instructions , this might satisfy your requirement

  1. Define the custom resource:

    qmgr –c "c r my_node_type type=string_array,flag=h"

  2. Append the custom resource to resources: line in sched_config file:
    resources: “ncpus, mem, arch, host, vnode, aoe, my_node_type”
    Note: after updating sched_config file, it is necessary to kill -HUP the pid of pbs scheduler.

  3. Set each queue’s default_chunk for the new resource to the value you are using
    to associate it with vnodes or compute nodes using qmgr:

    qmgr –c "s q redq default_chunk.my_node_type = red"
qmgr –c "s q blueq default_chunk.my_node_type = blue"
qmgr –c "s q greenq default_chunk.my_node_type = green"
qmgr –c "s q pinkq default_chunk.my_node_type = pink"

  4. Set the value for the new resource at each vnode:

    qmgr –c "s n node01 resources_available.my_node_type = red"
qmgr –c "s n node02 resources_available.my_node_type = blue"
qmgr –c "s n node03 resources_available.my_node_type = green"
qmgr –c "s n node04 resources_available.my_node_type = pink"

Note:

To submit jobs, so that jobs land only on node1:

  qsub -q redq -- /bin/sleep 100   # This job is scheduled on node01

To use cross utilization of nodes, append my_node_type to nodes (here we are using node01 / node02 as example)

  qmgr –c "s n node01 resources_available.my_node_type += blue"
  qmgr –c "s n node02 resources_available.my_node_type += red"

Now,

  qsub -q blueq -- /bin/sleep 100 # has two nodes node01 and node02 
  qsub -q redq  -- /bin/sleep 100 # has two nodes node01 and node02

I hope this helps

7

Thanks for the update. We have been also thinking on similar terms, on creating a “hostgroup” resource and then assign the desired value to it on each individual Host.

The scenario of selecting dynamically from a random list of Hosts does not seem to be supported in PBS. I do not have a good use case for this at the moment, except that some of our users who have been working on LSF are used to this feature.

For now, the single host option, and creating the “hostgroup” resource should suffice our needs.

Thanks,

8

By default, PBS Pro will select the nodes dynamically from the list of nodes (pbsnodes -av) available in the PBS complex. The scheduling policies , node / server configuration/limits are used to shepherd the jobs , so that maximum resource utilisation and usage politics are satisfied

Request to please explain the use cases and how that feature was used on your cluster. This can be helpful for all of us and our community members could address it.