Job is Not Running

Users/Site Administrators
1

Dear All,
The job is not running. Please help me to rectify this.

#PBS -N spicule
#PBS -l nodes=2:ppn=12
#PBS -l walltime=00:05:00
#PBS -q qreg_1day_small
###PBS -l mem=384gb
#PBS -o outtest.log
#PBS -e Error.log
cd /localscratch/physahel/pencil-code/sahel/spicule_strict_7.5G_small
export KNP_STACK_SIZE=100 G
rm data/svnid.dat
mpirun -machinefile $PBS_NODEFILE -np 24 ./src/start.x > start.out
mpirun -machinefile $PBS_NODEFILE -np 24 ./src/run.x > run.out
************************************************************
Currently Loaded Modulefiles:

  1. compilers/intel/2019.5.075 2) libs/git-2.9.5 3) compilers/mpich/3.1.3-int
    ************************************************************
    These outputs I got

phy@rnarasimha ]$ cat start.out

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= PID 190133 RUNNING AT node12
= EXIT CODE: 255
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= PID 143094 RUNNING AT node13
= EXIT CODE: 255
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES

[phy@rnarasimha ]$ cat run.out

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= PID 190148 RUNNING AT node12
= EXIT CODE: 255
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= PID 143212 RUNNING AT node13
= EXIT CODE: 255
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES

At the same time, I got these errors.

[phy@rnarasimha ]$ cat Error.log

[proxy:0:0@node12] HYDU_create_process (utils/launch/launch.c:75): execvp error on file ./src/start.x (No such file or directory)
[proxy:0:0@node12] HYDU_create_process (utils/launch/launch.c:75): execvp error on file ./src/start.x
[proxy:0:0@node12] HYDU_create_process (utils/launch/launch.c:75): execvp error on file ./src/run.x (No such file or directory)
[proxy:0:0@node12] HYDU_create_process (utils/launch/launch.c:75): execvp error on file ./src/run.x (No such file or directory)

2

Please update your script to include absolute path to the startx and runx and run it

#PBS -N spicule
#PBS -l select=2:ncpus=12:mpiprocs=12
#PBS -l walltime=00:05:00
#PBS -q qreg_1day_small
#PBS -o outtest.log
#PBS -e Error.log
cd /localscratch/physahel/pencil-code/sahel/spicule_strict_7.5G_small
export KNP_STACK_SIZE=100 G
rm data/svnid.dat
mpirun -machinefile $PBS_NODEFILE -np 24 /absolute/path/to/src/start.x > start.out
mpirun -machinefile $PBS_NODEFILE -np 24 /absolute/path/to/src/run.x > run.out

Also, please note the above errors are related to MPI layer. could you please try to run this batch command line on its own, creating the hosts.txt file and see whether it succeeds.
mpirun -machinefile hosts.txt -np 24 /absolute/path/to/src/start.x > start.out

3

Dear Adarsh,
I have successfully run a different setup under the same code. For that, I took 24 cores from 2 nodes. I got the problem with the current setup (spicule_strict_7.5G_small), which is for a bigger numerical domain. Although, I am successfully running the same setup in the different clusters.

Different setup under the same code.

#!/bin/bash
#PBS -N abc
#PBS -l nodes=2:ppn=12
#PBS -l walltime=00:05:00
#PBS -q qreg_1day_small
###PBS -l mem=12gb
#PBS -o outtest.log
#PBS -e Error.log
cd /localscratch/physahel/pencil-code/sahel/conv-slab
mpirun -machinefile $PBS_NODEFILE -np 24 ./src/start.x > start.out
mpirun -machinefile $PBS_NODEFILE -np 24 ./src/run.x > run.out

4

Please check this , hence try with absolute path in the pbs script for run.x
Also, please check the enviroment variable set on the compute nodes on the other cluster.
Try running mpirun with a readymade hostst.txt file and see whether it runs