How to fix job stay queued problem?

Users/Site Administrators
1

Dear PBS support team,
I got job queued when try to submit.

Here is log from /var/spool/PBS/server_logs:

07/24/2019 11:25:47;0100;Server@hpc;Req;;Type 0 request received from dtnguyen@hpc.private, sock=14
07/24/2019 11:25:47;0100;Server@hpc;Req;;Type 49 request received from dtnguyen@hpc.private, sock=15
07/24/2019 11:25:47;0100;Server@hpc;Req;;Type 21 request received from dtnguyen@hpc.private, sock=14
07/24/2019 11:25:47;0100;Server@hpc;Req;;Type 1 request received from dtnguyen@hpc.private, sock=14
07/24/2019 11:25:47;0086;Server@hpc;Svr;Server@hpc;Compiling script file: </var/spool/PBS/server_priv/hooks/PBS_ibwins.PY>
07/24/2019 11:25:47;0100;Server@hpc;Req;;Type 3 request received from dtnguyen@hpc.private, sock=14
07/24/2019 11:25:47;0100;Server@hpc;Req;;Type 5 request received from dtnguyen@hpc.private, sock=14
07/24/2019 11:25:47;0100;Server@hpc;Job;53.192.168.2.10;enqueuing into workq, state 1 hop 1
07/24/2019 11:25:47;0008;Server@hpc;Job;53.192.168.2.10;Job Queued at request of dtnguyen@hpc.private, owner = dtnguyen@hpc.private, job name = sleep_test, queue = workq
07/24/2019 11:25:47;0040;Server@hpc;Svr;192.168.2.10;Scheduler sent command 1
07/24/2019 11:25:47;0040;Server@hpc;Svr;192.168.2.10;Scheduler sent command 0
07/24/2019 11:25:47;0080;Server@hpc;Req;?;Req Header bad, errno 104, dis error 7
07/24/2019 11:25:47;0080;Server@hpc;Req;req_reject;Reject reply code=15056, aux=0, type=0, from @hpc.private
07/24/2019 11:25:47;0080;Server@hpc;Req;?;Req Header bad, errno 104, dis error 7
07/24/2019 11:25:47;0080;Server@hpc;Req;req_reject;Reject reply code=15056, aux=0, type=0, from @hpc.private
07/24/2019 11:25:52;0100;Server@hpc;Req;;Type 0 request received from dtnguyen@hpc.private, sock=15
07/24/2019 11:25:52;0100;Server@hpc;Req;;Type 49 request received from dtnguyen@hpc.private, sock=16
07/24/2019 11:25:52;0100;Server@hpc;Req;;Type 21 request received from dtnguyen@hpc.private, sock=15
07/24/2019 11:25:52;0100;Server@hpc;Req;;Type 19 request received from dtnguyen@hpc.private, sock=15

and no log show in compute node mom log.
my /etc/pbs.conf:
PBS_EXEC=/opt/pbs/default
PBS_HOME=/var/spool/PBS
PBS_START_SERVER=1
PBS_START_MOM=0
PBS_START_SCHED=1
PBS_START_COMM=1
PBS_LEAF_NAME=192.168.2.10
PBS_SERVER=192.168.2.10
PBS_CORE_LIMIT=unlimited
PBS_SCP=/usr/bin/scp

my pbs script file is so simple:
#!/bin/sh

#PBS -q workq
#PBS -o sleep.out.$PBS_JOBID
#PBS -e sleep.err.$PBS_JOBID
#PBS -N sleep_test
#PBS -l nodes=1:ppn=1

for i in {1..5} ; do
 echo $i
 sleep 5
done

and qmgr -c “p s” command:
#
# Create queues and set their attributes.
#
#
# Create and define queue workq
#
create queue workq
set queue workq queue_type = Execution
set queue workq enabled = True
set queue workq started = True
#
# Set server attributes.
#
set server scheduling = True
set server default_queue = workq
set server log_events = 511
set server mail_from = adm
set server query_other_jobs = True
set server resources_default.ncpus = 1
set server default_chunk.ncpus = 1
set server scheduler_iteration = 600
set server resv_enable = True
set server node_fail_requeue = 310
set server max_array_size = 10000
set server pbs_license_info = 6200@localhost
set server pbs_license_min = 1
set server pbs_license_max = 2147483647
set server pbs_license_linger_time = 31536000
set server license_count = Avail_Global:598 Avail_Local:1 Used:0 High_Use:0 Avail_Sockets:0 Unused_Sockets:0
set server eligible_time_enable = False
set server max_concurrent_provision = 5

Please help me solve that, thank you

2
  • The value of errno translates to “Connection reset by peer”. Suspect are something strange happening in the network or hostname resolution of the server . Maybe check for errors or dropped packets?
  • protocol mis-match if you are using different versions of PBS Pro within your complex

You do not have to use PBS_LEAF_NAME here, please remove it.
PBS_SERVER=<hostname that is reverse address resolvable and same across all /etc/hosts of headnode and compute node >

This is old syntax, it is supported, better to switch to new syntax
#PBS -l select=1:ncpus=1