# How to fix job stay queued problem?

**URL:** <https://community.openpbs.org/t/how-to-fix-job-stay-queued-problem/1711>\
**Category:** Users/Site Administrators\
**Created:** [July 24, 2019, 4:34am UTC](https://community.openpbs.org/t/how-to-fix-job-stay-queued-problem/1711 "2019-07-24T04:34:56Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![tungxp](https://avatars.discourse-cdn.com/v4/letter/t/8e7dd6/32.png) [@tungxp](https://community.openpbs.org/u/tungxp)\
**Post date:** [July 24, 2019, 4:34am UTC](https://community.openpbs.org/t/how-to-fix-job-stay-queued-problem/1711/1 "2019-07-24T04:34:56Z")

</div>

Dear PBS support team,  
I got job queued when try to submit.

Here is log from /var/spool/PBS/server\_logs:

> 07/24/2019 11:25:47;0100;Server@hpc;Req;;Type 0 request received from dtnguyen@hpc.private, sock=14  
> 07/24/2019 11:25:47;0100;Server@hpc;Req;;Type 49 request received from dtnguyen@hpc.private, sock=15  
> 07/24/2019 11:25:47;0100;Server@hpc;Req;;Type 21 request received from dtnguyen@hpc.private, sock=14  
> 07/24/2019 11:25:47;0100;Server@hpc;Req;;Type 1 request received from dtnguyen@hpc.private, sock=14  
> 07/24/2019 11:25:47;0086;Server@hpc;Svr;Server@hpc;Compiling script file: \</var/spool/PBS/server\_priv/hooks/PBS\_ibwins.PY\>  
> 07/24/2019 11:25:47;0100;Server@hpc;Req;;Type 3 request received from dtnguyen@hpc.private, sock=14  
> 07/24/2019 11:25:47;0100;Server@hpc;Req;;Type 5 request received from dtnguyen@hpc.private, sock=14  
> 07/24/2019 11:25:47;0100;Server@hpc;Job;53.192.168.2.10;enqueuing into workq, state 1 hop 1  
> 07/24/2019 11:25:47;0008;Server@hpc;Job;53.192.168.2.10;Job Queued at request of dtnguyen@hpc.private, owner = dtnguyen@hpc.private, job name = sleep\_test, queue = workq  
> 07/24/2019 11:25:47;0040;Server@hpc;Svr;192.168.2.10;Scheduler sent command 1  
> 07/24/2019 11:25:47;0040;Server@hpc;Svr;192.168.2.10;Scheduler sent command 0  
> 07/24/2019 11:25:47;0080;Server@hpc;Req;?;Req Header bad, errno 104, dis error 7  
> 07/24/2019 11:25:47;0080;Server@hpc;Req;req\_reject;Reject reply code=15056, aux=0, type=0, from @hpc.private  
> 07/24/2019 11:25:47;0080;Server@hpc;Req;?;Req Header bad, errno 104, dis error 7  
> 07/24/2019 11:25:47;0080;Server@hpc;Req;req\_reject;Reject reply code=15056, aux=0, type=0, from @hpc.private  
> 07/24/2019 11:25:52;0100;Server@hpc;Req;;Type 0 request received from dtnguyen@hpc.private, sock=15  
> 07/24/2019 11:25:52;0100;Server@hpc;Req;;Type 49 request received from dtnguyen@hpc.private, sock=16  
> 07/24/2019 11:25:52;0100;Server@hpc;Req;;Type 21 request received from dtnguyen@hpc.private, sock=15  
> 07/24/2019 11:25:52;0100;Server@hpc;Req;;Type 19 request received from dtnguyen@hpc.private, sock=15

and no log show in compute node mom log.  
my /etc/pbs.conf:  
PBS\_EXEC=/opt/pbs/default  
PBS\_HOME=/var/spool/PBS  
PBS\_START\_SERVER=1  
PBS\_START\_MOM=0  
PBS\_START\_SCHED=1  
PBS\_START\_COMM=1  
PBS\_LEAF\_NAME=192.168.2.10  
PBS\_SERVER=192.168.2.10  
PBS\_CORE\_LIMIT=unlimited  
PBS\_SCP=/usr/bin/scp

my pbs script file is so simple:  
#!/bin/sh

```
#PBS -q workq
#PBS -o sleep.out.$PBS_JOBID
#PBS -e sleep.err.$PBS_JOBID
#PBS -N sleep_test
#PBS -l nodes=1:ppn=1

for i in {1..5} ; do
 echo $i
 sleep 5
done

```

and qmgr -c “p s” command:  
#  
# Create queues and set their attributes.  
#  
#  
# Create and define queue workq  
#  
create queue workq  
set queue workq queue\_type = Execution  
set queue workq enabled = True  
set queue workq started = True  
#  
# Set server attributes.  
#  
set server scheduling = True  
set server default\_queue = workq  
set server log\_events = 511  
set server mail\_from = adm  
set server query\_other\_jobs = True  
set server resources\_default.ncpus = 1  
set server default\_chunk.ncpus = 1  
set server scheduler\_iteration = 600  
set server resv\_enable = True  
set server node\_fail\_requeue = 310  
set server max\_array\_size = 10000  
set server pbs\_license\_info = 6200@localhost  
set server pbs\_license\_min = 1  
set server pbs\_license\_max = 2147483647  
set server pbs\_license\_linger\_time = 31536000  
set server license\_count = Avail\_Global:598 Avail\_Local:1 Used:0 High\_Use:0 Avail\_Sockets:0 Unused\_Sockets:0  
set server eligible\_time\_enable = False  
set server max\_concurrent\_provision = 5

Please help me solve that, thank you

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [July 24, 2019, 7:17am UTC](https://community.openpbs.org/t/how-to-fix-job-stay-queued-problem/1711/2 "2019-07-24T07:17:36Z")

</div>

> [@tungxp](#):
>
> 07/24/2019 11:25:47;0080;Server@hpc;Req;?;Req Header bad, errno 104, dis error 7  
> 07/24/2019 11:25:47;0080;Server@hpc;Req;req\_reject;Reject reply code=15056, aux=0, type=0, from @hpc.private  
> 07/24/2019 11:25:47;0080;Server@hpc;Req;?;Req Header bad, errno 104, dis error 7  
> 07/24/2019 11:25:47;0080;Server@hpc;Req;req\_reject;Reject reply code=15056, aux=0, type=0, from @hpc.private

- The value of errno translates to “Connection reset by peer”. Suspect are something strange happening in the network or hostname resolution of the server . Maybe check for errors or dropped packets?
- protocol mis-match if you are using different versions of PBS Pro within your complex

> [@tungxp](#):
>
> PBS\_LEAF\_NAME=192.168.2.10  
> PBS\_SERVER=192.168.2.10

You do not have to use PBS\_LEAF\_NAME here, please remove it.  
PBS\_SERVER=\<hostname that is reverse address resolvable and same across all /etc/hosts of headnode and compute node \>

> [@Hostname issue setting up PBS Professional on CentOS 7](http://community.openpbs.org/t/hostname-issue-setting-up-pbs-professional-on-centos-7/1697/2):
>
> The value of PBS\_LEAF\_NAME should be the hostname of the interface over which you want PBS Pro to communicate, not the domain name. Unless you have multiple NICs, you should not need to set this value at all. Please try removing PBS\_LEAF\_NAME from /etc/pbs.conf and try starting PBS Pro.

> [@tungxp](#):
>
> #PBS -l nodes=1:ppn=1

This is old syntax, it is supported, better to switch to new syntax  
#PBS -l select=1:ncpus=1
