# Runjob hook modify select problem

**URL:** <https://community.openpbs.org/t/runjob-hook-modify-select-problem/268>\
**Category:** Users/Site Administrators\
**Created:** [September 29, 2016, 12:10pm UTC](https://community.openpbs.org/t/runjob-hook-modify-select-problem/268 "2016-09-29T12:10:01Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![buchmann](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/buchmann/32/31_2.png) [@buchmann](https://community.openpbs.org/u/buchmann)\
**Post date:** [September 29, 2016, 12:10pm UTC](https://community.openpbs.org/t/runjob-hook-modify-select-problem/268/1 "2016-09-29T12:10:01Z")

</div>

Dear Wizards,

I am writing hooks to deal with a reservation-like scenario.  
In certain cases, I will need to have a runtime hook reject a particular job (based on the node choice), and then force the job unto a specific node group (placement set).

According to the AG §6.10.5 table 6-13 (page AG-514) this should be possible (as long as I _reject_ the job), and it is stated that I should do something like:

```
jobB = pbs.event().job
jobB.select = pbs.select("2:mem=2gb:ncpus=1+6:mem=8gb:ncpus=16")

```

However, when I try to do this, the hook fails. (It does not fail when I import the hook, but rather at runtime).

The log states:

```
<date/server>;PBS server internal error (15011) in Error evaluating Python script, <class 'pbs.v1._exc_types.UnsetAttributeNameError'>
<date/server>;PBS server internal error (15011) in Error evaluating Python script, job attribute 'select' not found

```

I cannot say if this is a problem with the `job.select`, the documentation - or just that I did it wrong.  
For reference, my test code is simply (for debug purposes):

```
jobB=pbs.event().job
jobB.select = pbs.select("2:nodetype=io")
e.reject("Some comment")

```

All input is welcome! (Means “help!”)

Thanks,

Bjarne

---

<div class="post-metadata">

**Author:** ![scc](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/scc/32/234_2.png) [@scc](https://community.openpbs.org/u/scc)\
**Post date:** [September 29, 2016, 1:23pm UTC](https://community.openpbs.org/t/runjob-hook-modify-select-problem/268/2 "2016-09-29T13:23:15Z")

</div>

Hi Bjarne, yes, there is certainly a problem with that example in the documentation. It should show something like this:

```
jobB = pbs.event().job
jobB.Resource_List["place"] = pbs.place("pack:exclhost")
jobB.Resource_List["select"] = pbs.select("2:mem=2gb:ncpus=1+6:mem=8gb:ncpus=16")

```

So that would make your starting point something like this:

```
[root@centos7 tmp]# cat bjarne_r.py
import pbs
pbs.event().job.Resource_List["select"] = pbs.select("2:nodetype=io")
pbs.event().reject("Some comment")

```

This works with the caveat that the job must already have a nodetype request in the select statemt before the runjob hook tries to set it, otherwise it will not appear:

```
[root@centos7 tmp]# qmgr -c "s s scheduling=f"

[user1@centos7 ~]$ echo "echo bar" | qsub
11.centos7.prog.altair.com

[root@centos7 tmp]# qstat -f 11 | grep select
    Resource_List.select = 1:ncpus=1
    schedselect = 1:ncpus=1
[root@centos7 tmp]# qrun 11
qrun: Failed to run: Some comment (15136) 11.centos7.prog.altair.com
[root@centos7 tmp]# qstat -f 11 | grep select
    Resource_List.select = 1:ncpus=1
    schedselect = 1:ncpus=1
[root@centos7 tmp]# qalter -lselect=1:nodetype=compute 11
[root@centos7 tmp]# qstat -f 11 | grep select
    Resource_List.select = 1:nodetype=compute
    schedselect = 1:nodetype=compute:ncpus=1
[root@centos7 tmp]# qrun 11
qrun: Failed to run: Some comment (15136) 11.centos7.prog.altair.com
[root@centos7 tmp]# qstat -f 11 | grep select
    Resource_List.select = 2:nodetype=io
    schedselect = 2:nodetype=io:ncpus=1

```

I believe this is a bug and I will do more investigation and file one if need be.

Will all of your jobs already have a nodetype request as part of the job before run-time? If not, it is fairly trivial to create a queuejob hook which adds one and sets it to request a value like “any” (or whatever generic value you may already have in place in your configuration). Using a default\_chunk.nodetype server/queue attribute does not appear to be a viable workaround since it does not get applied to the job’s actual Resource\_List.select, but rather affects schedselect directly (which is what the scheduler ultimately actually uses to evaluate the job).

---

<div class="post-metadata">

**Author:** ![buchmann](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/buchmann/32/31_2.png) [@buchmann](https://community.openpbs.org/u/buchmann)\
**Post date:** [September 29, 2016, 1:27pm UTC](https://community.openpbs.org/t/runjob-hook-modify-select-problem/268/3 "2016-09-29T13:27:45Z")

</div>

The following seems to work (this is the AG example updated for the new work method):

```
jobB = pbs.event().job
jobB.Resource_List['select'] = pbs.select("2:mem=2gb:ncpus=1+6:mem=8gb:ncpus=16")

```

So, it seems the correct method to do this has been changed with the OSS update (PBS Pro v14), while the “newest doc” is still for v13.1. Probably, “`place`” should be equivalently updated.

It would be really good to get a list of the actual changes from v13 to v14 - especially when it comes to the API.

Best,

2\>Bjarne

---

<div class="post-metadata">

**Author:** ![buchmann](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/buchmann/32/31_2.png) [@buchmann](https://community.openpbs.org/u/buchmann)\
**Post date:** [September 29, 2016, 1:57pm UTC](https://community.openpbs.org/t/runjob-hook-modify-select-problem/268/4 "2016-09-29T13:57:03Z")

</div>

Hi Scott,

> [@scc](#):
>
> yes, there is certainly a problem with that example in the documentation

Thanks (apparently I was typing when your reply came in, so I did not see it). We agree on the solution.

> [@scc](#):
>
> Will all of your jobs already have a nodetype request as part of the job before run-time?

No. Some of the jobs will have it set and some not. The hook should not try to change the set. In certain cases (when the high-priority nodes are “reserved”), the hook will add a low-priority nodetype=io, but it will not change it. However, I need to make another hook, which can remove the added nodetype=io, just in case the job has not yet run, when the reservation is lifted. (In certain cases, the job will not be able to run on the io-nodes, but will just remain queued - which in our case is ok for a while).

> [@scc](#):
>
> This works with the caveat that the job must already have a nodetype request in the select statemt before the runjob hook tries to set it

Hmmm. That is not what I see.  
Here is my example case (with comments added along the way).

```
[bjb@bifrost1 ~]$ qsub -l walltime=00:02:00 -l select=2:ncpus=2 -l place=scatter:excl -- /bin/sleep 30
1386.bifrost1

[bjb@bifrost1 ~]$ qstat2lin 1386|grep -e comment -e select
    Resource_List.select = 2:ncpus=2
    schedselect = 2:ncpus=2
    Submit_arguments = -l walltime=00:02:00 -l select=2:ncpus=2 -l place=scatter:excl -- /bin/sleep 30

```

`qstat2lin` is just my hack for `qstat -f` without the line breaks, see  
[Qstat -f \<JOBID\> line breaks - #2 by buchmann](http://community.openpbs.org/t/qstat-f-jobid-line-breaks/267/2)  
[DCOO / ClusterTools / [faa9dc] /pbs/qstat2lin](https://sourceforge.net/p/dcoo/cluster-tools/ci/master/tree/pbs/qstat2lin)

The following will let the scheduler try to run the job. The hook will catch it, and update `select` - forcing it onto the io-nodes:

```
[bjb@bifrost1 ~]$ qmgr -c 'set server scheduling = 1'

```

Note that both `Resource_List.select` and `schedselect ` get updated:

```
[bjb@bifrost1 ~]$ qstat2lin 1386|grep -e comment -e select
    Resource_List.select = 2:ncpus=2:nodetype=io
    schedselect = 2:ncpus=2:nodetype=io
    comment = Not Running: PBS Error: JOBSELECT2io - nodetype=io explicitly added to job select, as the walltime overlaps the reserve time set by the adminstrator
    Submit_arguments = -l walltime=00:02:00 -l select=2:ncpus=2 -l place=scatter:excl -- /bin/sleep 30

```

Luckily `Submit_arguments` is not updated, so in principle, I can use that (match for nodetype=io) to see if the hooks have fiddled with the nodetype. Alternativelty, I might try to store something in `job.Variable_List`. (I cannot rely on the `comment` field, as the server/scheduler may overwrite what I have put in.

Forcing yet another scheduling iteration, and the scheduler realizes that I really have only defined _one_ io-node, so the job cannot really run:

```
[bjb@bifrost1 ~]$ qmgr -c 'set server scheduling = 1'
[bjb@bifrost1 ~]$ /home/bjb/DcooWare/dcoo-clustertools-ssh2016-09-27/pbs/qstat2lin 1386|grep -e comment -e select
    Resource_List.select = 2:ncpus=2:nodetype=io
    schedselect = 2:ncpus=2:nodetype=io
    comment = Can Never Run: can't fit in the largest placement set, and can't span psets
    Submit_arguments = -l walltime=00:02:00 -l select=2:ncpus=2 -l place=scatter:excl -- /bin/sleep 30

```

My nice comment got overwritten, but really the behaviour of the system is exactly what I expect.

I have tried also using `qrun` and it does not seem to make a difference: nodetype still gets correctly set. So, with my usage I **can’t** confirm “that the job must already have a nodetype request in the select statemt before the runjob hook tries to set it”.

Thank you so much for answering!

/Bjarne

---

<div class="post-metadata">

**Author:** ![scc](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/scc/32/234_2.png) [@scc](https://community.openpbs.org/u/scc)\
**Post date:** [September 29, 2016, 2:14pm UTC](https://community.openpbs.org/t/runjob-hook-modify-select-problem/268/5 "2016-09-29T14:14:44Z")

</div>

Hi Bjarne, you are right, it is not about the job having requested a nodetype in a select statement in particular, rather the critical difference is whether or not the job requested ANYTHING in a select statement explicitly:

```
[user1@centos7 ~]$ echo "echo bar" | qsub -lselect=1:ncpus=1
16.centos7.prog.altair.com

[root@centos7 tmp]# qstat -f 16 | grep select
    Resource_List.select = 1:ncpus=1
    schedselect = 1:ncpus=1
    Submit_arguments = -lselect=1:ncpus=1
[root@centos7 tmp]# qrun 16
qrun: Failed to run: Some comment (15136) 16.centos7.prog.altair.com
[root@centos7 tmp]# qstat -f 16 | grep select
    Resource_List.select = 2:nodetype=io
    schedselect = 2:nodetype=io:ncpus=1
    Submit_arguments = -lselect=1:ncpus=1

[user1@centos7 ~]$ echo "echo bar" | qsub
17.centos7.prog.altair.com

[root@centos7 tmp]# qstat -f 17 | grep select
    Resource_List.select = 1:ncpus=1
    schedselect = 1:ncpus=1
[root@centos7 tmp]# qrun 17
qrun: Failed to run: Some comment (15136) 17.centos7.prog.altair.com
[root@centos7 tmp]# qstat -f 17 | grep select
    Resource_List.select = 1:ncpus=1
    schedselect = 1:ncpus=1
```

---

<div class="post-metadata">

**Author:** ![scc](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/scc/32/234_2.png) [@scc](https://community.openpbs.org/u/scc)\
**Post date:** [September 29, 2016, 8:00pm UTC](https://community.openpbs.org/t/runjob-hook-modify-select-problem/268/6 "2016-09-29T20:00:16Z")

</div>

PP-470 has been filed for the inability to set a select statement for a job which was submitted without one.

An internal Altair ticket has been filed for the documentation bug.

Thanks!

---

<div class="post-metadata">

**Author:** ![buchmann](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/buchmann/32/31_2.png) [@buchmann](https://community.openpbs.org/u/buchmann)\
**Post date:** [September 29, 2016, 8:01pm UTC](https://community.openpbs.org/t/runjob-hook-modify-select-problem/268/7 "2016-09-29T20:01:29Z")

</div>

> [@scc](#):
>
> the critical difference is whether or not the job requested ANYTHING in a select statement

So, if the hook find the select part empty, then it is (with the present version of the code) not a good idea to add (anything) to the select, right?  
Yes, that might be considered a bug. However, if `-lresource` is used (it could be if - explicitly or not - if `-lselect` is not used), then we would probably not be allowed to update `select`. In that case it could probably be OK for the server to throw away any updates to `select`?

Best,

/Bjarne

---

<div class="post-metadata">

**Author:** ![scc](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/scc/32/234_2.png) [@scc](https://community.openpbs.org/u/scc)\
**Post date:** [September 29, 2016, 10:08pm UTC](https://community.openpbs.org/t/runjob-hook-modify-select-problem/268/8 "2016-09-29T22:08:28Z")

</div>

Actually if -lresource (or -lnodes for that matter) are used then the server translates them into a select statement for the job. You DO have to worry about whether the submitter used -lselect or -lncpus or -lnodes when writing a queuejob hook since that translation has not taken place yet at queuejob time, but that is not a concern for a runjob hook.

---

<div class="post-metadata">

**Author:** ![buchmann](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/buchmann/32/31_2.png) [@buchmann](https://community.openpbs.org/u/buchmann)\
**Post date:** [September 30, 2016, 5:33am UTC](https://community.openpbs.org/t/runjob-hook-modify-select-problem/268/9 "2016-09-30T05:33:26Z")

</div>

Thanks a bunch, Scott.

> [@scc](#):
>
> the server translates them into a select statement for the job. … that translation has not taken place yet at queuejob time

That is really helpful advice. I see that, writing to `select` for a `queuejob` hook could lead to problems.  
I guess that might be mentioned in the AG(v13.1) §6.10.5 incl. Table 6-13, where it appears that the “Job `select` statement” can be read and set (`r, s`) by **queuejob** hooks. There seems to be no special caveats associated with this. It would be very useful information to add in this section, if there are special cases, where (for instance) `select` should/may/can not be updated.

/Bjarne

PS: On a side note, I wonder about this text on p. AG-514 (v13.1) §6.10.5 (also related to Table 6-3):

> [@](#):
>
> The following table shows … an “o” indicates that this value can be set but the action has no effect.

However, there are no occurrences of “o” in the table(?)

---

<div class="post-metadata">

**Author:** ![agurban](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/agurban/32/67_2.png) [@agurban](https://community.openpbs.org/u/agurban)\
**Post date:** [October 3, 2016, 9:29pm UTC](https://community.openpbs.org/t/runjob-hook-modify-select-problem/268/10 "2016-10-03T21:29:40Z")

</div>

Hi Bjarne,

I’ve filed a doc bug for this issue. Thank you for noticing this.

Regards,  
-Anne