I’ve used python as my submission script in the past rather than bash etc. In slurm conder I usually instructed the path of the compiler at the top. I tried it in qsub but it didn’t work. Any idea how to make it work?
#/homes/miranda9/.conda/envs/automl-meta-learning/lib/python3.7 #PBS -V #PBS -M email@example.com #PBS -m abe import sys for p in sys.path: print(p)
$ cat test_qsub.py.e381299 /homes/miranda9/.profile: line 2: /opt/intel/compilers_and_libraries_2017/linux/mpi/intel64/bin/mpivars.sh: No such file or directory /homes/miranda9/.profile: line 3: /opt/intel/compilers_and_libraries_2017/linux/bin/compilervars.sh: No such file or directory import: unable to open X server `' @ error/import.c/ImportImageCommand/369. /var/spool/pbs/mom_priv/jobs/381299.iam-pbs.SC: line 9: syntax error near unexpected token `print' /var/spool/pbs/mom_priv/jobs/381299.iam-pbs.SC: line 9: ` print(p)'
after I added
! and made ‘#!/homes/miranda9/.conda/envs/automl-meta-learning/lib/python3.7’ at the top I got a new error:
$ cat test_qsub.py.e381301 /homes/miranda9/.profile: line 2: /opt/intel/compilers_and_libraries_2017/linux/mpi/intel64/bin/mpivars.sh: No such file or directory /homes/miranda9/.profile: line 3: /opt/intel/compilers_and_libraries_2017/linux/bin/compilervars.sh: No such file or directory -bash: /var/spool/pbs/mom_priv/jobs/381301.iam-pbs.SC: /homes/miranda9/.conda/envs/automl-meta-learning/lib/python3.7: bad interpreter: Permission denied
- How does one use python as a submission script using the PBS scheduler?
not sure why the reply I pasted is not appearing but here is my soln:
First don’t forget the
! at the top.
The trick is to figure out where the binary for python is (with your conda environment for example) and tell PBS at the top for your submission script. I tried using
sys.path but that lead to a path to a folder that PBS couldn’t use. I had to find out the
sys.executable to find it. Then I just put it at the top.
Demo session to find it:
$ python Python 3.7.7 (default, Mar 26 2020, 15:48:22) [GCC 7.3.0] :: Anaconda, Inc. on linux Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.executable '/homes/miranda9/.conda/envs/myenv/bin/python'
then place it properly with the exclamation mark:
#!/homes/miranda9/.conda/envs/YOURENV/bin/python #PBS -V #PBS -M firstname.lastname@example.org #PBS -m abe #PBS -lselect=1:ncpus=112 import sys import os for p in sys.path: print(p) print(os.environ)