Hello,
I have a hook that works normal until the python_restart_max_hooks value is reached. I understand Python Interpreter will restart when that and python_restart_min_interval are reached.
The problem is, that the hook stops working after that, and all the jobs are rejected because the hook does not run (hook type is queuejob).
In order to solve this I systemctl restart pbs.
Is there a way to get the hook working back to normal without restarting pbs?
PBS Version 22.05.11
This is the logs when I applied systemctl restart pbs
08/18/2022 17:15:22;0100;Server@testkm;Svr;Server@testkm;–> Python Interpreter started, compiled with version:'3.6.8 (default, Apr 12 2022, 06:55:39)
08/18/2022 17:15:22;0100;Server@testkm;Svr;Server@testkm;sys.modules= {‘builtins’: <module ‘builtins’ (built-in)>, ‘sys’: <module ‘sys’ (built-in)>, ‘_frozen_importlib’: <module ‘_frozen_importlib’ (frozen)>, ‘_imp’: <module ‘_imp’ (built-in)>, ‘_warnings’: <module ‘_warnings’ (built-in)>, ‘_thread’: <module ‘_thread’ (built-in)>, ‘_weakref’: <module ‘_weakref’ (built-in)>, ‘_frozen_importlib_external’: <module ‘_frozen_importlib_external’ (frozen)>, ‘_io’: <module ‘io’ (built-in)>, ‘marshal’: <module ‘marshal’ (built-in)>, ‘posix’: <module ‘posix’ (built-in)>, ‘zipimport’: <module ‘zipimport’ (built-in)>, ‘encodings’: <module ‘encodings’ from ‘/usr/lib64/python3.6/encodings/init.py’>, ‘codecs’: <module ‘codecs’ from ‘/usr/lib64/python3.6/codecs.py’>, ‘_codecs’: <module ‘_codecs’ (built-in)>, ‘encodings.aliases’: <module ‘encodings.aliases’ from ‘/usr/lib64/python3.6/encodings/aliases.py’>, ‘encodings.ascii’: <module ‘encodings.ascii’ from ‘/usr/lib64/python3.6/encodings/ascii.py’>, ‘_signal’: <module ‘_signal’ (built-in)>, ‘main’: <module ‘main’ (built-in)>, ‘encodings.utf_8’: <module ‘encodings.utf_8’ from ‘/usr/lib64/python3.6/encodings/utf_8.py’>, ‘encodings.latin_1’: <module ‘encodings.latin_1’ from ‘/usr/lib64/python3.6/encodings/latin_1.py’>, ‘io’: <module ‘io’ from ‘/usr/lib64/python3.6/io.py’>, ‘abc’: <module ‘abc’ from ‘/usr/lib64/python3.6/abc.py’>, ‘_weakrefset’: <module ‘_weakrefset’ from ‘/usr/lib64/python3.6/_weakrefset.py’>, ‘pbs’: <module ‘pbs’ from ‘/opt/pbs/lib/python/altair/pbs/init.py’>, ‘_pbs_v1’: <module ‘_pbs_v1’ (built-in)>, ‘pbs.v1’: <module ‘pbs.v1’ from ‘/opt/pbs/lib/python/altair/pbs/v1/init.py’>, ‘pbs.v1._export_types’: <module ‘pbs.v1._export_types’ from ‘/opt/pbs/lib/python/altair/pbs/v1/_export_types.py’>, ‘pbs.v1._base_types’: <module ‘pbs.v1._base_types’ from ‘/opt/pbs/lib/python/altair/pbs/v1/_base_types.py’>, ‘math’: <module ‘math’ from ‘/usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so’>, ‘pbs.v1._exc_types’: <module ‘pbs.v1._exc_types’ from ‘/opt/pbs/lib/python/altair/pbs/v1/_exc_types.py’>, ‘pbs.v1._svr_types’: <module ‘pbs.v1._svr_types’ from ‘/opt/pbs/lib/python/altair/pbs/v1/_svr_types.py’>, ‘os’: <module ‘os’ from ‘/usr/lib64/python3.6/os.py’>, ‘errno’: <module ‘errno’ (built-in)>, ‘stat’: <module ‘stat’ from ‘/usr/lib64/python3.6/stat.py’>, ‘_stat’: <module ‘_stat’ (built-in)>, ‘posixpath’: <module ‘posixpath’ from ‘/usr/lib64/python3.6/posixpath.py’>, ‘genericpath’: <module ‘genericpath’ from ‘/usr/lib64/python3.6/genericpath.py’>, ‘os.path’: <module ‘posixpath’ from ‘/usr/lib64/python3.6/posixpath.py’>, ‘_collections_abc’: <module ‘_collections_abc’ from ‘/usr/lib64/python3.6/_collections_abc.py’>, ‘pbs.v1._pmi_types’: <module ‘pbs.v1._pmi_types’ from ‘/opt/pbs/lib/python/altair/pbs/v1/_pmi_types.py’>, ‘pbs.v1._pmi_utils’: <module ‘pbs.v1._pmi_utils’ from ‘/opt/pbs/lib/python/altair/pbs/v1/_pmi_utils.py’>}
This is what happens when python_restart_max_hooks and python_restart_min_interval are reached.
08/18/2022 17:15:22;0100;Server@testkm;Hook;_pbs_python_event_set;Current memory usage: VmSize=293720kB, VmRSS=52152kB
08/18/2022 17:15:22;0001;Server@testkm;Svr;Server@testkm;PBS server internal error (15011) in Error evaluating Python script, <class ‘ImportError’>
08/18/2022 17:15:22;0001;Server@testkm;Svr;Server@testkm;PBS server internal error (15011) in Error evaluating Python script, cannot import name ‘encodings’
Much appreciatted your support