Use of python/hooks with PBS

Users/Site Administrators
1

Dear Wizards,

I am trying to figure out how to use/write hooks to interface with PBSpro.

Before I start PBS-integration, I would like to see that I can actually use the pbs library. So, initially, I simply write a python script, which use the pbs module as per:

try:
  import pbs
except:
  print "Failed to import pbs"

I have encountered a few issues so far.

a. The admin guide (v13, AG-647) mention that the modules should be in PBS_EXEC/python and PBS_EXEC/lib/python. In my case, /opt/pbs/python does not exist.

b. If I add “/opt/pbs/lib/python” to the python path (append to sys.path - or modify PYTHONPATH, then

import pbs

results in a compile-time error

ImportError: No module named pbs

From the tree structure of /opt/pbs/lib it appears that a new dir level “altair” has been introduced

[root]# find /opt/pbs/lib -type d
/opt/pbs/lib
/opt/pbs/lib/MPI
/opt/pbs/lib/init.d
/opt/pbs/lib/python
/opt/pbs/lib/python/altair
/opt/pbs/lib/python/altair/pbs
/opt/pbs/lib/python/altair/pbs/v1
/opt/pbs/lib/python/altair/pbs_hooks

So, if I add “/opt/pbs/python/altair” to the python path, then I should be able to access the module(?)
However, although the pbs module is found in this case, the script still fails at the use statement - now with the error:

[bjb@bifrost1 PythonTest]$ ./test_pbspy.py
Traceback (most recent call last):
  File "./test_pbspy.py", line 10, in <module>
    import pbs
  File "/opt/pbs/lib/python/altair/pbs/__init__.py", line 43, in <module>
    from _pbs_v1 import *
ImportError: No module named _pbs_v1

I am not sure what I am doing wrong. But it must be something really basic.

If somebody could point me in the right direction, then I’ll be grateful.

Thanks,

/Bjarne

PS:
I have installed pbspro v 14.1 on CentOS72 - compiled to rpms locally, with a few changes to the spec file, as mentioned in a previous thread CentOS7 install issues

2

Hi Bjarne,
I have never tried to compile the hook external to pbs so not sure if that can be done. You can import your python file to a hook in pbs and it would get compiled successfully when pbs runs it.

eg.

qmgr -c "create hook test_hook event=queuejob"
qmgr -c "import hook test_hook application/x-python default path_to_python_hook_script"
qmgr -c “set hook test_hook enabled=true”

Regards
Dilip

3

Hello,

PBS runs hooks using pbs_python command (wrapper over system python) and this takes care of setting up the environment before executing the script. So as Dilip mentioned you could create a hook using qmgr command and import your script to PBS.

Do you have any specific use case to run your scripts external to PBS?

Thanks,
Ashwath

4

Simple testing and debugging - before entering into a more complicated environment. KISS.

/Bjarne

5

Hi Bjarne,

To try out “import pbs”, you’ll need to use the ‘pbs_python’ command provided by PBS. Use of ‘python’ by itself will not work. Please check v13 Admin Guide section 6 on “Debugging Hooks”. On a basic level, simply create an input file, say ‘hook.input’ with the content:

pbs.event().type=queuejob

and then a sample python hook scipt, say “hook.py” with content:

import pbs
e=pbs.event()
print e.type
And run pbs_python with the --hook option as follows:

% pbs_python --hook -i hook.input hook.py
1

If you leave out ‘hook.py’ file, then it runs in interactive mode:

% pbs_python --hook -i hook.in

Python 2.5.1 (r251:54863, Feb 2 2015, 15:12:04)
[GCC 4.3.2 [gcc-4_3-branch revision 141291]] on linux2
>> import pbs
>> e=pbs.event()
>> print e.type
1

>> e.accept() <-- to exit out of the prompt

1 Like
6

Thanks,

I have tried this out - with a simple input/event file as per the man page for pbs_python.
It seems to work fine.

One comment - but that is related to the man page.

<man>
 ...
 Example input file:
             pbs.event().hook_name=proto
             pbs.event().hook_type=site
             pbs.event().type=queuejob
             pbs.event().requestor=user1
             pbs.event().requestor_host=host1
             pbs.event().hook_alarm=40
             pbs.event().job.id=72
             pbs.event().job.Job_Name=job1
             pbs.event().job.Resource_List[ncpus]=5
             pbs.event().job.Resource_List[mem]=6mb
             pbs.event().vnode_list["host1"].resources_available["ncpus"] = 5
             pbs.event().vnode_list["host1"].resources_available["mem"] = 300gb
 ...
</man>

The line with “hook_alarm” caused an error

pbs_python: unknown event attribute 'hook_alarm'

Presumably “hook_alarm” is just an old param no longer supported. As you point out there are also good examples in the admin guide.

Apart from this little issue, this has set me on the right path. Thanks again.

/Bjarne

7

Bjarne,

Good to know you got it working! For the error, it should be just 'pbs.event().alarm=".

Although, when I tried it, and access the value in the hook script via ‘print pbs.event().alarm’, I get an EventIncompatbileError’ exception. I’ll file a bug on this and also have our documentation fixed.

Update: Filed bug PP-417 (cannot reference pbs.event().alarm in standalone pbs_python) on this.