# Jobs stuck in Q stat and the scheduler seems not working right

**URL:** <https://community.openpbs.org/t/jobs-stuck-in-q-stat-and-the-scheduler-seems-not-working-right/2371>\
**Category:** Users/Site Administrators\
**Created:** [November 20, 2020, 1:03am UTC](https://community.openpbs.org/t/jobs-stuck-in-q-stat-and-the-scheduler-seems-not-working-right/2371 "2020-11-20T01:03:46Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![yaowen](https://avatars.discourse-cdn.com/v4/letter/y/45deac/32.png) [@yaowen](https://community.openpbs.org/u/yaowen)\
**Post date:** [November 20, 2020, 1:03am UTC](https://community.openpbs.org/t/jobs-stuck-in-q-stat-and-the-scheduler-seems-not-working-right/2371/1 "2020-11-20T01:03:46Z")

</div>

Hi all,

I just installed the openpbs in a single server following the github INSTALL instructions. Everything looks fine except that all jobs stuck in Q state. I did some search and have no idea how to fix it. I guess the scheduler is not working right. Below is something may help to find any clue. Thank you for your time to help me.

$ /etc/init.d/pbs status  
pbs\_server is pid 10064  
pbs\_mom is pid 9871  
pbs\_sched is pid 9884  
pbs\_comm is 9861

$cat /etc/pbs.conf  
PBS\_SERVER=thirteen  
PBS\_START\_SERVER=1  
PBS\_START\_SCHED=1  
PBS\_START\_COMM=1  
PBS\_START\_MOM=1  
PBS\_EXEC=/opt/pbs  
PBS\_HOME=/var/spool/pbs  
PBS\_CORE\_LIMIT=unlimited  
PBS\_SCP=/bin/scp

$cat /etc/hosts  
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4  
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6  
10.9.9.13 thirteen

$ifconfig  
…  
enp28s0: flags=4163\<UP,BROADCAST,RUNNING,MULTICAST\> mtu 1500  
inet 10.9.9.13 netmask 255.255.255.0 broadcast 10.9.9.255  
inet6 fe80::a3f9:6640:2012:c4ae prefixlen 64 scopeid 0x20  
ether e8:61:1f:29:90:66 txqueuelen 1000 (Ethernet)  
RX packets 69849 bytes 24324069 (23.1 MiB)  
RX errors 0 dropped 2785 overruns 0 frame 0  
TX packets 4643 bytes 1959986 (1.8 MiB)  
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0  
device memory 0xaae00000-aae1ffff  
…

$hostname -f  
thirteen

sestatus SELinux status: disabled nmap thirteen  
Starting Nmap 6.40 ( [http://nmap.org](http://nmap.org) ) at 2020-11-20 08:53 CST  
Nmap scan report for thirteen (10.9.9.13)  
Host is up (0.0000050s latency).  
Not shown: 996 closed ports  
PORT STATE SERVICE  
22/tcp open ssh  
111/tcp open rpcbind  
15002/tcp open unknown  
15003/tcp open unknown

Nmap done: 1 IP address (1 host up) scanned in 0.12 seconds

$firewall-cmd --state  
not running

$pbs\_hostn -v thirteen  
primary name: thirteen (from gethostbyname())  
aliases: -none-  
address length: 4 bytes  
address: 10.9.9.13 (218695946 dec) name: thirteen

$qstat -a  
thirteen:  
Req’d Req’d Elap  
Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time

* * *

8.thirteen achen workq STDIN – 1 1 – – Q –  
9.thirteen achen workq STDIN – 1 1 – – Q –  
…

$ps -ef | grep pbs\_  
root 9861 1 0 08:42 ? 00:00:00 /opt/pbs/sbin/pbs\_comm  
root 9871 1 0 08:42 ? 00:00:00 /opt/pbs/sbin/pbs\_mom  
root 9884 1 0 08:42 ? 00:00:00 /opt/pbs/sbin/pbs\_sched  
root 9966 1 0 08:42 ? 00:00:00 /opt/pbs/sbin/pbs\_ds\_monitor monitor  
postgres 10063 10006 0 08:42 ? 00:00:00 postgres: postgres pbs\_datastore 10.9.9.13(49620) idle  
root 10064 1 0 08:42 ? 00:00:00 /opt/pbs/sbin/pbs\_server.bin  
root 11208 9398 0 08:55 pts/2 00:00:00 grep --color=auto pbs\_

$ systemctl status pbs

- pbs.service - Portable Batch System  
Loaded: loaded (/opt/pbs/libexec/pbs\_init.d; enabled; vendor preset: disabled)  
Active: inactive (dead) since Fri 2020-11-20 08:28:12 CST; 28min ago  
Docs: man:pbs(8)  
Process: 7479 ExecStop=/opt/pbs/libexec/pbs\_init.d stop (code=exited, status=0/SUCCESS)  
Process: 1657 ExecStart=/opt/pbs/libexec/pbs\_init.d start (code=exited, status=0/SUCCESS)

Nov 20 07:37:03 thirteen su[2466]: (to postgres) root on none  
Nov 20 07:37:06 thirteen su[2520]: (to postgres) root on none  
Nov 20 07:37:17 thirteen pbs\_init.d[1657]: Starting PBS in background  
Nov 20 08:28:06 thirteen su[7229]: (to postgres) root on none  
Nov 20 08:28:07 thirteen su[7262]: (to postgres) root on none  
Nov 20 08:28:10 thirteen su[7298]: (to postgres) root on none  
Nov 20 08:28:10 thirteen pbs\_init.d[7479]: Stopping PBS  
Nov 20 08:28:11 thirteen su[7539]: (to postgres) root on none  
Nov 20 08:28:11 thirteen su[7577]: (to postgres) root on none  
Nov 20 08:28:11 thirteen pbs\_init.d[7479]: Waiting for shutdown to complete

$pbsnodes -av  
thirteen  
Mom = thirteen  
Port = 15002  
pbs\_version = 20.0.0  
ntype = PBS  
state = free  
pcpus = 48  
resources\_available.arch = linux  
resources\_available.host = thirteen  
resources\_available.mem = 329653792kb  
resources\_available.ncpus = 48  
resources\_available.vnode = thirteen  
resources\_assigned.accelerator\_memory = 0kb  
resources\_assigned.hbmem = 0kb  
resources\_assigned.mem = 0kb  
resources\_assigned.naccelerators = 0  
resources\_assigned.ncpus = 0  
resources\_assigned.vmem = 0kb  
resv\_enable = True  
sharing = default\_shared  
license = l  
last\_state\_change\_time = Fri Nov 20 08:56:41 2020

$ qstat -answ1

thirteen:  
Req’d Req’d Elap  
Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time

* * *

8.thirteen achen workq STDIN – 1 1 – – Q – –  
–  
9.thirteen achen workq STDIN – 1 1 – – Q – –

$ ping thirteen  
PING thirteen (10.9.9.13) 56(84) bytes of data.  
64 bytes from thirteen (10.9.9.13): icmp\_seq=1 ttl=64 time=0.020 ms  
64 bytes from thirteen (10.9.9.13): icmp\_seq=2 ttl=64 time=0.019 ms  
…

$nslookup thirteen  
Server: 8.8.8.8  
Address: 8.8.8.8#53

\*\* server can’t find thirteen: NXDOMAIN

$cat /var/spool/pbs/sched\_logs/20201120  
…  
11/20/2020 08:56:35;0002;pbs\_sched;Svr;Log;Log opened  
11/20/2020 08:56:35;0002;pbs\_sched;Svr;pbs\_sched;pbs\_version=20.0.0  
11/20/2020 08:56:35;0002;pbs\_sched;Svr;pbs\_sched;pbs\_build=mach=N/A:security=N/A:configure\_args=N/A  
11/20/2020 08:56:35;0002;pbs\_sched;Svr;pbs\_sched;hostname=thirteen;pbs\_leaf\_name=N/A;pbs\_mom\_node\_name=N/A  
11/20/2020 08:56:35;0002;pbs\_sched;Svr;pbs\_sched;ipv4 interface lo: localhost4.localdomain4  
11/20/2020 08:56:35;0002;pbs\_sched;Svr;pbs\_sched;ipv4 interface enp28s0: thirteen  
11/20/2020 08:56:35;0002;pbs\_sched;Svr;pbs\_sched;ipv6 interface lo: localhost6.localdomain6  
11/20/2020 08:56:35;0002;pbs\_sched;Svr;pbs\_sched;ipv6 interface enp28s0: thirteen  
11/20/2020 08:56:35;0002;pbs\_sched;n/a;setup\_env;read environment from /var/spool/pbs/pbs\_environment  
11/20/2020 08:56:35;0006;pbs\_sched;Fil;pbs\_sched;Version 20.0.0, started, initialization type = 0  
11/20/2020 08:56:35;0002;pbs\_sched;Svr;main;/opt/pbs/sbin/pbs\_sched startup pid 11644  
11/20/2020 08:56:35;0040;pbs\_sched;Fil;sched\_config;Error reading line 398:  
11/20/2020 08:56:35;0040;pbs\_sched;Fil;fairshare usage;Creating usage database for fairshare  
11/20/2020 08:56:35;0080;pbs\_sched;Req;;Launching 24 worker threads  
11/20/2020 08:56:39;0001;pbs\_sched;Svr;pbs\_sched;Access from host not allowed, or unknown host (15008) in connect\_svrpool, Couldn’t register the scheduler default with the configured servers  
11/20/2020 08:56:41;0001;pbs\_sched;Svr;pbs\_sched;Access from host not allowed, or unknown host (15008) in connect\_svrpool, Couldn’t register the scheduler default with the configured servers  
11/20/2020 08:56:43;0001;pbs\_sched;Svr;pbs\_sched;Access from host not allowed, or unknown host (15008) in connect\_svrpool, Couldn’t register the scheduler default with the configured servers  
11/20/2020 08:56:46;0001;pbs\_sched;Svr;pbs\_sched;Access from host not allowed, or unknown host (15008) in connect\_svrpool, Couldn’t register the scheduler default with the configured servers  
11/20/2020 08:56:48;0001;pbs\_sched;Svr;pbs\_sched;Access from host not allowed, or unknown host (15008) in connect\_svrpool, Couldn’t register the scheduler default with the configured servers  
11/20/2020 08:56:50;0001;pbs\_sched;Svr;pbs\_sched;Access from host not allowed, or unknown host (15008) in connect\_svrpool, Couldn’t register the scheduler default with the configured servers  
11/20/2020 08:56:52;0001;pbs\_sched;Svr;pbs\_sched;Access from host not allowed, or unknown host (15008) in connect\_svrpool, Couldn’t register the scheduler default with the configured servers  
11/20/2020 08:56:54;0001;pbs\_sched;Svr;pbs\_sched;Access from host not allowed, or unknown host (15008) in connect\_svrpool, Couldn’t register the scheduler default with the configured servers  
11/20/2020 08:56:56;0001;pbs\_sched;Svr;pbs\_sched;Access from host not allowed, or unknown host (15008) in connect\_svrpool, Couldn’t register the scheduler default with the configured servers  
11/20/2020 08:56:58;0001;pbs\_sched;Svr;pbs\_sched;Access from host not allowed, or unknown host (15008) in connect\_svrpool, Couldn’t register the scheduler default with the configured servers  
…

##### 

By the way, I have to submit job using another account rather than root. Using root account to submit jobs leads to a BAD UID error.

Hope anyone could help. Please let me know if further details need to be posted here.

---

<div class="post-metadata">

**Author:** ![mkaro](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/mkaro/32/85_2.png) [@mkaro](https://community.openpbs.org/u/mkaro)\
**Post date:** [November 20, 2020, 4:50pm UTC](https://community.openpbs.org/t/jobs-stuck-in-q-stat-and-the-scheduler-seems-not-working-right/2371/2 "2020-11-20T16:50:55Z")

</div>

Looks like some recent changes may be causing problems. Did you use the master branch? This can be a bit unstable at times. Best to pull from one of the tags here: [https://github.com/openpbs/openpbs/tags](https://github.com/openpbs/openpbs/tags)

The tags represent stable release points.

---

<div class="post-metadata">

**Author:** ![yaowen](https://avatars.discourse-cdn.com/v4/letter/y/45deac/32.png) [@yaowen](https://community.openpbs.org/u/yaowen)\
**Post date:** [November 21, 2020, 2:24am UTC](https://community.openpbs.org/t/jobs-stuck-in-q-stat-and-the-scheduler-seems-not-working-right/2371/3 "2020-11-21T02:24:45Z")

</div>

Thanks @mkaro for your kind reply.

I tried the 20.0.1 version, and have two further problems .  
First, how could I uninstall the older version of open-pbs installed from source? I just re-install the 20.0.1 version and the pbs service could not start. Below is the message:  
$/etc/init.d/pbs start  
Starting PBS  
PBS Home directory /var/spool/pbs needs updating.  
Running /opt/pbs/libexec/pbs\_habitat to update it.

* * *

## Cannot upgrade PBS datastore version 1.5.0 Failed to upgrade PBS Datastore

I guess it’s because I didn’t uninstall the older version. Could you please give me some advice?

Second, I tried to install version 20.0.1 into another server. When I followed the exactly same instructions to install, I met a difficult problem:  
$ make  
…  
libtool: link: gcc -g -O2 -o pbs\_server.bin pbs\_server\_bin-accounting.o pbs\_server\_bin-array\_func.o pbs\_server\_bin-attr\_recov.o pbs\_server\_bin-attr\_recov\_db.o pbs\_server\_bin-checkkey.o pbs\_server\_bin-dis\_read.o pbs\_server\_bin-failover.o pbs\_server\_bin-geteusernam.o pbs\_server\_bin-hook\_func.o pbs\_server\_bin-issue\_request.o pbs\_server\_bin-job\_func.o pbs\_server\_bin-job\_recov.o pbs\_server\_bin-job\_recov\_db.o pbs\_server\_bin-job\_route.o pbs\_server\_bin-license\_client.o pbs\_server\_bin-mom\_info.o pbs\_server\_bin-node\_func.o pbs\_server\_bin-node\_manager.o pbs\_server\_bin-node\_recov\_db.o pbs\_server\_bin-pbsd\_init.o pbs\_server\_bin-pbsd\_main.o pbs\_server\_bin-process\_request.o pbs\_server\_bin-queue\_func.o pbs\_server\_bin-queue\_recov\_db.o pbs\_server\_bin-reply\_send.o pbs\_server\_bin-req\_delete.o pbs\_server\_bin-req\_getcred.o pbs\_server\_bin-req\_holdjob.o pbs\_server\_bin-req\_jobobit.o pbs\_server\_bin-req\_locate.o pbs\_server\_bin-req\_manager.o pbs\_server\_bin-req\_message.o pbs\_server\_bin-req\_modify.o pbs\_server\_bin-req\_preemptjob.o pbs\_server\_bin-req\_movejob.o pbs\_server\_bin-req\_quejob.o pbs\_server\_bin-req\_register.o pbs\_server\_bin-req\_rerun.o pbs\_server\_bin-req\_rescq.o pbs\_server\_bin-req\_runjob.o pbs\_server\_bin-req\_select.o pbs\_server\_bin-req\_shutdown.o pbs\_server\_bin-req\_signal.o pbs\_server\_bin-req\_stat.o pbs\_server\_bin-req\_track.o pbs\_server\_bin-req\_cred.o pbs\_server\_bin-resc\_attr.o pbs\_server\_bin-resv\_attr.o pbs\_server\_bin-run\_sched.o pbs\_server\_bin-sched\_func.o pbs\_server\_bin-setup\_resc.o pbs\_server\_bin-stat\_job.o pbs\_server\_bin-svr\_attr.o pbs\_server\_bin-svr\_chk\_owner.o pbs\_server\_bin-svr\_connect.o pbs\_server\_bin-svr\_func.o pbs\_server\_bin-svr\_jobfunc.o pbs\_server\_bin-svr\_mail.o pbs\_server\_bin-svr\_migrate\_data.o pbs\_server\_bin-svr\_movejob.o pbs\_server\_bin-svr\_recov\_db.o pbs\_server\_bin-svr\_resccost.o pbs\_server\_bin-svr\_credfunc.o pbs\_server\_bin-user\_func.o pbs\_server\_bin-vnparse.o …/…/src/lib/Libtpp/libtpp.a …/…/src/lib/Libattr/libattr.a …/…/src/lib/Libutil/libutil.a …/…/src/lib/Liblog/liblog.a …/…/src/lib/Libnet/libnet.a …/…/src/lib/Libsec/libsec.a …/…/src/lib/Libsite/libsite.a …/…/src/lib/Libpython/libpbspython\_svr.a …/…/src/lib/Libdb/libdb.a …/…/src/lib/Libpbs/.libs/libpbs.a -lpq /home/achen/anaconda3/lib/libexpat.so -L/home/achen/anaconda3/lib -lz -lical -L/home/achen/anaconda3/lib/python3.7/config-3.7m-x86\_64-linux-gnu -lpython3.7m -lpthread -lutil -lrt -lm -lssl -lcrypto -ldl -lcrypt -lc -Wl,-rpath -Wl,/home/achen/anaconda3/lib -Wl,-rpath -Wl,/home/achen/anaconda3/lib  
/usr/bin/ld: warning: libssl.so.10, needed by /usr/lib/gcc/x86\_64-redhat-linux/4.8.5/…/…/…/…/lib64/libpq.so, may conflict with libssl.so.1.1  
/usr/bin/ld: warning: libcrypto.so.10, needed by /usr/lib/gcc/x86\_64-redhat-linux/4.8.5/…/…/…/…/lib64/libpq.so, may conflict with libcrypto.so.1.1  
/usr/bin/ld: …/…/src/lib/Libpbs/.libs/libpbs.a(libpbs\_la-pbs\_aes\_encrypt.o): undefined reference to symbol ‘EVP\_CIPHER\_CTX\_init@@libcrypto.so.10’  
/usr/lib64/libcrypto.so.10: error adding symbols: DSO missing from command line  
collect2: error: ld returned 1 exit status  
make[2]: \*\*\* [pbs\_server.bin] Error 1  
make[2]: Leaving directory `/home/achen/openpbs-20.0.1/src/server' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/home/achen/openpbs-20.0.1/src’  
make: \*\*\* [all-recursive] Error 1

Hope you could give me some advice, thanks a lot!

---

<div class="post-metadata">

**Author:** ![yaowen](https://avatars.discourse-cdn.com/v4/letter/y/45deac/32.png) [@yaowen](https://community.openpbs.org/u/yaowen)\
**Post date:** [November 25, 2020, 3:13am UTC](https://community.openpbs.org/t/jobs-stuck-in-q-stat-and-the-scheduler-seems-not-working-right/2371/4 "2020-11-25T03:13:23Z")

</div>

Could you please give me some advice when you have time? 😭 Thanks~

---

<div class="post-metadata">

**Author:** ![mkaro](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/mkaro/32/85_2.png) [@mkaro](https://community.openpbs.org/u/mkaro)\
**Post date:** [November 25, 2020, 3:29pm UTC](https://community.openpbs.org/t/jobs-stuck-in-q-stat-and-the-scheduler-seems-not-working-right/2371/5 "2020-11-25T15:29:07Z")

</div>

If this is a new installation, and you don’t need to preserve anything from previous installations, the easiest thing to do is to stop PBS and then remove /var/spool/pbs completely. Upon restart, PBS will run the pbs\_habitat script to recreate the necessary directories and files. I think the reason you are seeing a problem is that the DB upgrade script only works when migrating from an older version to a newer version of the database, and you are doing the opposite by reverting back to 20.0.1.

If you have settings or jobs you need to preserve, my suggestion is to drain the queues and to capture the output of: qmgr -c “print server” \> /tmp/my\_settings

This can then be fed back into qmgr when the 20.0.1 instance is started: cat /tmp/my\_settings | qmgr

---

<div class="post-metadata">

**Author:** ![yaowen](https://avatars.discourse-cdn.com/v4/letter/y/45deac/32.png) [@yaowen](https://community.openpbs.org/u/yaowen)\
**Post date:** [December 1, 2020, 5:11am UTC](https://community.openpbs.org/t/jobs-stuck-in-q-stat-and-the-scheduler-seems-not-working-right/2371/6 "2020-12-01T05:11:26Z")

</div>

@mkaro Thank you very much for your kind help.

---

<div class="post-metadata">

**Author:** ![austin](https://avatars.discourse-cdn.com/v4/letter/a/77aa72/32.png) [@austin](https://community.openpbs.org/u/austin)\
**Post date:** [December 6, 2022, 1:37am UTC](https://community.openpbs.org/t/jobs-stuck-in-q-stat-and-the-scheduler-seems-not-working-right/2371/7 "2022-12-06T01:37:34Z")

</div>

Hi did you resolve the issue?

I seem to have the same problem. My is version `pbs_version = 20.0.0`.

Lemme know if and how you addressed it. Thanks.