Error using the "run_epilogue_shell hook "

Users/Site Administrators
1

Hello OpenPBS community team,

I have OpenPBS 23.06 installed on Ubuntu 22.04.
Setting up for Epilogue processing.
I can run it properly in /var/spool/pbs/mom_priv/epilogue script, but via Hook, I get the following error and cannot run it

03/12/2024 10:07:30;0008;pbs_mom;Job;1032.kvm-pbs;Started, pid = 9039
03/12/2024 10:07:45;0008;pbs_mom;Job;1032.kvm-pbs;Terminated
03/12/2024 10:07:45;0008;pbs_mom;Job;1032.kvm-pbs;kill_job
03/12/2024 10:07:45;0001;pbs_mom;Svr;pbs_mom;scan_for_exiting, chdir failed. ERR : Permission denied
03/12/2024 10:07:45;0006;pbs_python;Hook;pbs_python;run_epilogue_shell;1032.kvm-pbs;running epilogue.
03/12/2024 10:07:46;0001;pbs_python;Svr;pbs_python;PBS server internal error (15011) in Error evaluating Python script, <class 'AttributeError'>
03/12/2024 10:07:46;0001;pbs_python;Svr;pbs_python;PBS server internal error (15011) in Error evaluating Python script, 'AttributeError' object has no attribute 'message'
03/12/2024 10:07:46;0001;pbs_mom;Svr;pbs_mom;run_hook, execv of /opt/pbs/bin/pbs_python resulted in nonzero exit status=254
03/12/2024 10:07:46;0008;pbs_mom;Job;1032.kvm-pbs;execjob_epilogue hook 'run_epilogue_shell' encountered an exception, request rejected
03/12/2024 10:07:46;0008;pbs_mom;Job;1032.kvm-pbs;no active tasks
03/12/2024 10:07:47;0001;pbs_mom;Svr;pbs_mom;fork_to_user, chdir failed. ERR : Permission denied
03/12/2024 10:07:47;0008;pbs_mom;Job;1032.kvm-pbs;no active tasks
03/12/2024 10:07:47;0008;pbs_mom;Job;1032.kvm-pbs;kill_job

I have set it up in the following way

create hook run_epilogue_shell
set hook run_epilogue_shell event = execjob_epilogue
set hook run_epilogue_shell enabled = true
set hook run_epilogue_shell order = 999
set hook run_prologue_shell alarm = 35
import hook run_epilogue_shell application/x-python default run_pelog_shell.py
import hook run_epilogue_shell application/x-config default run_pelog_shell.ini``

Any advice would be appreciated!
Please let me know if you need more information

Regards,
Hiromi

2

Please share the execjob_epilogue hook.

3

adarsh,

Thank you for your reply.

execjob_epilogue is:

I think the problem is in the following areas

    trace = {
        'line': sys.exc_info()[2].tb_lineno,
        'module': sys.exc_info()[2].tb_frame.f_code.co_name,
        'exception': sys.exc_info()[0].__name__,
        'message': sys.exc_info()[1].message,
    }

Correct as follows:

    trace = {
        'line': sys.exc_info()[2].tb_lineno,
        'module': sys.exc_info()[2].tb_frame.f_code.co_name,
        'exception': sys.exc_info()[0].__name__,
        'message': sys.exc_info()[1]
    }

What do you think? Is there a problem?

Hiromi,

4

Thank you @hiromi
Could you please share the run_pelog_shell.ini

5

adarsh,

ini file is following,

No changes made to scripts or ini files

Thanks！

6

[run_pelog_shell]
ENABLE_PARALLEL=False
VERBOSE_USER_OUTPUT=False
DEFAULT_ACTION=RERUN
TORQUE_COMPAT=False

Thank you @hiromi
Could you please try using only attribute lines without any comments and see whether it makes difference

7

I removed the commented lines from the hook script and ran it, but the result did not change

8

I changed the Line 197 of run_pelog_shell.py
‘message’: sys.exc_info()[1].message,.
->‘message’: sys.exc_info()[1].

This fix eliminates the python error, but there seems to be other problems with this Hook.

I would like to coexist with cgroup’s Hook and run the epilogue script at the end of the job. I do not intend to stick to this Hook in particular. If there is a simpler way, I would like to try it.

Any advice on how to run the epilogue script with pbs_cgroups Hook ?

9

When executing an epilogue script that outputs to stdout, an error occurred with the default settings.

04/03/2024 14:45:55;0006;pbs_python;Hook;pbs_python;run_epilogue_shell;6.kvm-pbs;running epilogue.
04/03/2024 14:45:56;0006;pbs_python;Hook;pbs_python;execjob_epilogue hook run_epilogue_shell encountered an exception: Line 247 in stdout PermissionError: [Errno 13] Permission denied: ‘/var/spool/pbs/spool/6.kvm-pbs.OU’
04/03/2024 14:45:56;0008;pbs_mom;Job;6.kvm-pbs;Hook Error: request rejected as filter hook ‘run_epilogue_shell’ encountered an exception. Please inform Admin

I was able to work around this by changing the hook user to pbsuser.
I don’t know why I can’t write to the stdout file as root, but I was able to run the epilogue for the time being!

Thank you very much.