# Problems with resources\_max.nodes

**URL:** <https://community.openpbs.org/t/problems-with-resources-max-nodes/208>\
**Category:** Users/Site Administrators\
**Created:** [August 11, 2016, 1:28pm UTC](https://community.openpbs.org/t/problems-with-resources-max-nodes/208 "2016-08-11T13:28:20Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![buchmann](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/buchmann/32/31_2.png) [@buchmann](https://community.openpbs.org/u/buchmann)\
**Post date:** [August 11, 2016, 1:28pm UTC](https://community.openpbs.org/t/problems-with-resources-max-nodes/208/1 "2016-08-11T13:28:20Z")

</div>

I cant seem to enforce a maximum restriction/limit on the number of nodes allowed for a single job.

If I am not mistaken, this could be done either on queue or server level, by configuring (in qmgr):

sudo qmgr -c "set queue prod resources\_max.nodes = 3"  
and/or  
sudo qmgr -c “set server resources\_max.nodes = 3”

However in any case (even with both enforced), the system happily starts jobs with 4 or more nodes as per:  
qsub -I -l nodes=4

I have tried to strip down the config to a bare minimum, but the problem persists.  
Do any of you know if this feature is working?

From the logs, I can gather that the job requests and gets ncpus=1 on each of 4 nodes:

LOG: Job Run at request of [Scheduler@bifrost1.fcoo.dk](mailto:Scheduler@bifrost1.fcoo.dk) on exec\_vnode (dn008:ncpus=1)+(dn101:ncpus=1)+(dn102:ncpus=1)+(dn103:ncpus=1)

And it is “counted” as just resources\_used.ncpus=4 at the end:

LOG: Exit\_status=0 resources\_used.cpupercent=0 resources\_used.cput=00:00:00 resources\_used.mem=0kb resources\_used.ncpus=4 resources\_used.vmem=0kb resources\_used.walltime=00:00:02

Each vnode has ncpus=20, but even specifying “4 vnodes with 20 cpus each” is allowed to run:

qsub -I -l select=4:ncpus=20

I guess I am doing _something_ wrong, although I cannot figure it out. For sure it feels like the resources\_max.nodes is somehow broken. If so, is there an alternative?

Thanks,

/Bjarne

---

<div class="post-metadata">

**Author:** ![dilip-krishnan](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/dilip-krishnan/32/75_2.png) [@dilip-krishnan](https://community.openpbs.org/u/dilip-krishnan)\
**Post date:** [August 11, 2016, 3:31pm UTC](https://community.openpbs.org/t/problems-with-resources-max-nodes/208/2 "2016-08-11T15:31:00Z")

</div>

Hi Bjarne,  
Can you try setting limit using resources\_max.nodect=3

---

<div class="post-metadata">

**Author:** ![bhroam](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bhroam/32/20_2.png) [@bhroam](https://community.openpbs.org/u/bhroam)\
**Post date:** [August 11, 2016, 10:52pm UTC](https://community.openpbs.org/t/problems-with-resources-max-nodes/208/3 "2016-08-11T22:52:06Z")

</div>

Hey Bjarne,  
The reason setting resources\_max.nodes did not work is that the nodes resource is a string. Setting resources\_max on a string will only do an exact string match. Dilip is correct that setting resources\_max.nodect should work for you.

As a warning, nodect is really a chunk count now. This won’t make a difference if you use the old ‘nodes’ syntax(or -lplace=scatter). If you start using the select syntax, resources\_max.nodect may not work as desired. A job requesting -lselect=4:ncpus=1 will have a nodect=4. This job will be rejected on submission due to the resources\_max. The scheduler would have happily run all 4 chunks on one vnode.

Bhroam

---

<div class="post-metadata">

**Author:** ![buchmann](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/buchmann/32/31_2.png) [@buchmann](https://community.openpbs.org/u/buchmann)\
**Post date:** [August 12, 2016, 6:20am UTC](https://community.openpbs.org/t/problems-with-resources-max-nodes/208/4 "2016-08-12T06:20:43Z")

</div>

Thanks, guys. This helps a lot.

> [@dilip-krishnan](#):
>
> Can you try setting limit using resources\_max.nodect=3

That does the trick. Jobs with -l nodes=3 run fine, while nodes=4 are refused by with

> qsub: Job violates queue and/or server resource limits  
> (qsub exits with code 188).

> [@bhroam](#):
>
> As a warning, nodect is really a chunk count now.

I can confirm this. A job with, say, -l select=3:ncpus=1 is accepted and scheduled to run on a single node (20 cores on the node), while -l select=4:ncpus=1 is refused before even making it to the scheduler.

Thumbs up to both of you! Kudos!

/Bjarne