Bug: job object crippled in modifyjob hooks?

Users/Site Administrators
1

Dear Wizards,

I have (yet another) issue with hooks.
I have defined a “holding queue” - an execution queue, which is enabled but not started - where I can put jobs which are not allowed to run for a while. A cron-based script will pull the jobs out of the queue at some later time.

The problem is that my hooks (say, a modifyjob hook) cannot seem to see the actual queue name. When they run, pbs.event().job.queue seems to be None. So, I cannot do “my special stuff” to the jobs.

Is this normal/expected. And if so, is there a way, so can get the same information inside the hook execution?

qstat clearly shows the queue as holdq.

Thanks,

/Bjarne

2

This seems to be an issue not with the queue state (not started) but with the hook type (modifyhook).

I trigger a modifyjob hook issuing a qalter -a <TIM> <JID> with a time specification (TIM) 30 secs in the future. Alternatively, by specifying a user list (qalter -u luser <JID>)
In either case, when the modifyjob hook executes, it can’t read eg. job queue, state and comment. It can read job id, and write eg. job comment.

Sample hook code:

pbs.logmsg(pbs.LOG_DEBUG, "Executing %s for jobid=%s"%(hookname,pbs.event().job.id))
if pbs.event().job.queue == None:
  pbs.logmsg(pbs.LOG_DEBUG, "  Cannot access job queue")
if pbs.event().job.job_state == None:
  pbs.logmsg(pbs.LOG_DEBUG, "  Cannot access job state")
if pbs.event().job.comment == None:
  pbs.logmsg(pbs.LOG_DEBUG, "  Cannot access job comment")

Sample server log:

<date/server>;Executing reserve_hold_modifyjob for jobid=1496.bifrost1
<date/server>;  Cannot access job queue
<date/server>;  Cannot access job state
<date/server>;  Cannot access job comment

[bjb@bifrost1 ~]$ qstat -f 1496.bifrost1  | grep -e comment -e queue -e state
    job_state = Q
    queue = workq
    substate = 10
    queue_rank = 934
    queue_type = E
    comment = Hook reserve_hold_modifyjob was here

I have no clue what I may be doing wrong. All input is more than welcome!

2>Bjarne

3

In a modifyjob hook, if I get the job object as described in the AG (v13.1, p633, §16.3), :

e = pbs.event()
j = e.job

then I get into all the issues above - as I do not seem to get the “actual”/“full” job object.
However, If I then just use this initial job object, and ask the server for the “real” job ojbect (with matching ID), like:

j2=pbs.server().job( j.id )
#j2=pbs.server().job( pbs.event().job.id )

Edit: Then I get read access to all properties of the job.
However, for j2 I than cannot update, say, comment with:

j2.comment = "My new comment"

The server log then reports:

<date/server>;PBS server internal error (15011) in Error evaluating Python script, <class 'pbs.v1._exc_types.BadAttributeValueError'>
<date/server>;PBS server internal error (15011) in Error evaluating Python script, attribute 'comment' is part of a readonly object

So, I still need to keep the event job object around, so I can do modifications.

With this work-around, I can now write my modifyjob hook, but I would certainly consider this a bug, unless somebody can explain to me what goes on.

Best,

/Bjarne

EDIT: PS: Updated/changed the title of this thread to better represent the actual issue.

4

Hello Bjarne, please see this section of the PBS Pro documentation, I think this is the root of all of these problems/questions:

A pbs.MODIFYJOB event has the following members, in addition to those listed in Table 6-17, “Using Event Object Members in Events,” on page 572 and Table 6-28, “Methods Available in Events,” on page 609:

pbs.event().job
A pbs.job object representing the job being modified. See section 6.12.7, “Job Objects”, on page 589. This job object contains only those attributes and resources specified for modification. This job object does not contain any attributes or resources that are not to be modified.

pbs.event().job_o
A pbs.job object representing the original job, before the job was modified via qalter. See section 6.12.4.16.vii, “Original Job Event Member”, on page 575.

A pbs.event().accept() terminates hook execution and allows the job to be altered, and any changes to job attributes or resources take effect.
A pbs.event().reject() terminates hook execution and causes the job not to be altered.

In other words, look at pbs.event().job_o for the job attributes going into the hook execution, and change what you need to change on pbs.event().job.

I hope this helps.

-Scott

5

Hi Scott,

Thank you very much for the explanation. I think we are (at most) talking about a documentation issue.

That is very good to know, and it is - for sure - the crucial info related to my issues.
I would probably phrase it as “This job object only contains the attributes and resources that are to be modified by the qalter command.”
Obviously, there are a few extra things in the object, such as the Variable_List dictionary, but I assume that this does not qualify as “attribute or resource”.

I have been over the documentation yet again. And I must say that it is still not apparent to me how I should reached the conclusion that pbs.event().job should not be a “full” job. It surely is nice that a modifyjob hook has a way to know what features were updated. But I still feel befuddled.

§6.12.4.16.v (p 575) clearly states without caveats that pbs.event().job is “A pbs.job object.”. I read this as a “full” object (I still have difficulties interpreting differently) and so it appears on the pages describing the job object (pp589ff). As you write, p 575 also specifies pbs.event().job_o as how the job looked before the modification - but not that this should bear impact on the “other” job object. I do not see that on p 589 either:

Again, I do not read that modifyjob hooks have a “special” version of pbs.event().job.

To wrap it up: With your explanation, the implementation seems fine. However, in the documentation (AG §6.12.4.16.v p 575) regarding pbs.event().job, I could have used an explicit mentioning that this object is special for the modifyjob hooks.

Sure did. Thanks again, Scott.

/Bjarne

PS: Quite likely, job_o is just the same object as what I get from asking the server - a read-only version of the “full” job - before any alteration. Obviously, I should code it as job_o, rather than what I wrote above.