How does one use python as a submission script using the PBS scheduler?

Developers
1

I’ve used python as my submission script in the past rather than bash etc. In slurm conder I usually instructed the path of the compiler at the top. I tried it in qsub but it didn’t work. Any idea how to make it work?

Code:

#/homes/miranda9/.conda/envs/automl-meta-learning/lib/python3.7
#PBS -V
#PBS -M brando.wontputithere@gmail.com
#PBS -m abe

import sys

for p in sys.path:
    print(p)

error msg:

$ cat test_qsub.py.e381299 
/homes/miranda9/.profile: line 2: /opt/intel/compilers_and_libraries_2017/linux/mpi/intel64/bin/mpivars.sh: No such file or directory
/homes/miranda9/.profile: line 3: /opt/intel/compilers_and_libraries_2017/linux/bin/compilervars.sh: No such file or directory
import: unable to open X server `' @ error/import.c/ImportImageCommand/369.
/var/spool/pbs/mom_priv/jobs/381299.iam-pbs.SC: line 9: syntax error near unexpected token `print'
/var/spool/pbs/mom_priv/jobs/381299.iam-pbs.SC: line 9: `    print(p)'

after I added ! and made ‘#!/homes/miranda9/.conda/envs/automl-meta-learning/lib/python3.7’ at the top I got a new error:

$ cat test_qsub.py.e381301 
/homes/miranda9/.profile: line 2: /opt/intel/compilers_and_libraries_2017/linux/mpi/intel64/bin/mpivars.sh: No such file or directory
/homes/miranda9/.profile: line 3: /opt/intel/compilers_and_libraries_2017/linux/bin/compilervars.sh: No such file or directory
-bash: /var/spool/pbs/mom_priv/jobs/381301.iam-pbs.SC: /homes/miranda9/.conda/envs/automl-meta-learning/lib/python3.7: bad interpreter: Permission denied

crossposted:

not sure why the reply I pasted is not appearing but here is my soln:

First don’t forget the ! at the top.

The trick is to figure out where the binary for python is (with your conda environment for example) and tell PBS at the top for your submission script. I tried using sys.path but that lead to a path to a folder that PBS couldn’t use. I had to find out the sys.executable to find it. Then I just put it at the top.

Demo session to find it:

$ python
Python 3.7.7 (default, Mar 26 2020, 15:48:22) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.executable
'/homes/miranda9/.conda/envs/myenv/bin/python'

then place it properly with the exclamation mark:

#!/homes/miranda9/.conda/envs/YOURENV/bin/python
#PBS -V
#PBS -M youremail@gmail.com
#PBS -m abe
#PBS -lselect=1:ncpus=112

import sys
import os

for p in sys.path:
    print(p)

print(os.environ)

works now!

2

First don’t forget the ! at the top.

The trick is to figure out where the binary for python is (with your conda environment for example) and tell PBS at the top for your submission script. I tried using sys.path but that lead to a path to a folder that PBS couldn’t use. I had to find out the sys.executable to find it. Then I just put it at the top.

Demo session to find it:

$ python
Python 3.7.7 (default, Mar 26 2020, 15:48:22) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.executable
'/homes/miranda9/.conda/envs/myenv/bin/python'

then place it properly with the exclamation mark:

#!/homes/miranda9/.conda/envs/YOURENV/bin/python
#PBS -V
#PBS -M youremail@gmail.com
#PBS -m abe
#PBS -lselect=1:ncpus=112

import sys
import os

for p in sys.path:
    print(p)

print(os.environ)

works now!