OS: RHEL9.7 5.14.0-611.47.1.el9_7.x86_64
BCM 11 (NVIDIA Base Command Manager)
OpenPBS 23.06.06
We upgraded our BCM 10 cluster on RHEL8 to BCM 11 on RHEL9. After the upgrade our OpenPBS jobs are failing and in the MOM logs we see the following error repeatedly:
04/21/2026 10:19:33;0080;pbs_python;Hook;pbs_python;Unexpected error: syntax error: line 1, column 0
I’ve looked this up and this error suggests that “pbs_python” was compiled against a different version of Python3 than is being used, but pbs_python is part of the BCM installation which also includes both all of the OpenPBS stuff and its own integrated Python3, so I would not expect to see this.
-– RHEL Python3 —
[root@torhpcplp11 ~]# which python3
/usr/bin/python3
[root@torhpcplp11 ~]# /usr/bin/python3 -V
Python 3.9.25
-– BCM Python3 —
[hooedd@torhpcplp16(surfside) ~]$ module load python312
[root@torhpcplp11 ~]# module load python3
[root@torhpcplp11 ~]# which python3
/cm/local/apps/python3/bin/python3
[root@torhpcplp11 ~]# /cm/local/apps/python312/bin/python3 -V
Python 3.12.13
-– pbs_python —
[root@torhpcplp11 ~]# /cm/shared/apps/openpbs/23.06.06/bin/pbs_python -V
Python 3.9.25
[root@torhpcplp11 ~]# /cm/shared/apps/openpbs/23.06.06/bin/pbs_python --version
pbs_version = 23.06.06
-– On compute nodes —
[root@cpu002 ~]# which python3
/usr/bin/python3
[root@cpu002 ~]# /usr/bin/python3 -V
Python 3.9.25