Dear Dale, All,
I have run the ‘journalctl’ + ‘dmesg’ commands on that node and definitely there was not any issues related to OOM or segfault, that is why I have grabbed only the relevant ‘pbs_mom’ related output.
Since the cluster is shared by many users and different workflows we definitely need pbs_hook in our running environment. However as soon as the ‘improper sid’ error is encountered it affects only the new jobs, all other which were already running on the affected node, keep going so “update_job_usage” is very normal.
Yesterday we had a system maintenance window for the entire cluster (new image, but the pbs_mom still the same) and as result all compute nodes have been rebooted. So far I have not seen any ‘improper sid’ error messages, but at the same time I lost access to all old ‘dmesg’ and journalctl.
I will keep an eye and as soon as I have something new I will update that post.
On the other hand what you would suggest ?
strace -p pbs_mom_ID -f -o /tmp/ouput_of_strace (any other options?)
or attach dbg to a running mom ? To narrow down the issue .
Again thanks…
All the best
Roman