# Jobs stuck in queue, need to use root access to qrun to initiate each job **URL:** \ **Category:** Users/Site Administrators\ **Created:** [December 22, 2022, 10:29pm UTC](https://community.openpbs.org/t/jobs-stuck-in-queue-need-to-use-root-access-to-qrun-to-initiate-each-job/3396 "2022-12-22T22:29:28Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![wamclaughlin](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/wamclaughlin/32/619_2.png) [@wamclaughlin](https://community.openpbs.org/u/wamclaughlin)\ **Post date:** [December 22, 2022, 10:29pm UTC](https://community.openpbs.org/t/jobs-stuck-in-queue-need-to-use-root-access-to-qrun-to-initiate-each-job/3396/1 "2022-12-22T22:29:28Z")
Dear All, I recently installed OpenPBS version 20.0.0 and am currently trying to overcome a problem with scheduling my submitted jobs. I was able to create a queue (named batch), and submit jobs to this queue, however, the jobs will stay queued unless I force them to run with the “qrun” command as root user. I did check my system requirements, and the problem is not arising from this. I am seeking someone’s help to determine what could be done to fix this issue. My overall goal is to send jobs to the queue and have the job scheduler complete these jobs as system requirements become available. Please find below some details regarding my system. We are running OpenPBS on a single server. OUTPUT of etc/hosts: 10.18.62.60 cc-3dfr.bcrc.local cc-3dfr -------------------------------------------------------------------------------------------------------OUTPUT of etc/pbs.conf PBS\_SERVER=cc-3dfr PBS\_START\_SERVER=1 PBS\_START\_SCHED=1 PBS\_START\_COMM=1 PBS\_START\_MOM=1 PBS\_EXEC=/opt/pbs PBS\_HOME=/var/spool/pbs PBS\_CORE\_LIMIT=unlimited PBS\_SCP=/usr/bin/scp -------------------------------------------------------------------------------------------------------OUTPUT of pbsnodes –av (base) [modeleval@cc-3dfr etc]$ pbsnodes -av cc-3dfr Mom = cc-3dfr.bcrc.local ntype = PBS state = free pcpus = 24 resources\_available.arch = linux resources\_available.host = cc-3dfr resources\_available.mem = 148244804kb resources\_available.ncpus = 24 resources\_available.vnode = cc-3dfr resources\_assigned.accelerator\_memory = 0kb resources\_assigned.hbmem = 0kb resources\_assigned.mem = 0kb resources\_assigned.naccelerators = 0 resources\_assigned.ncpus = 0 resources\_assigned.vmem = 0kb resv\_enable = True sharing = default\_shared license = l last\_state\_change\_time = Tue Nov 29 03:49:03 2022 last\_used\_time = Wed Dec 21 14:33:06 2022 ------------------------------------------------------------------------------------------------------ADDITIONAL COMMANDS: (base) [modeleval@cc-3dfr ~]$ hostname cc-3dfr (base) [modeleval@cc-3dfr ~]$ hostname -f cc-3dfr.bcrc.local (base) [modeleval@cc-3dfr etc]$ pbs\_hostn -v cc-3dfr primary name: cc-3dfr.bcrc.local (from gethostbyname()) aliases: cc-3dfr address length: 4 bytes address: 10.18.62.60 (1514672650 dec) name: cc-3dfr.bcrc.local (base) [modeleval@cc-3dfr ~]$ ping cc-3dfr PING cc-3dfr.bcrc.local (10.18.62.60) 56(84) bytes of data. 64 bytes from cc-3dfr.bcrc.local (10.18.62.60): icmp\_seq=1 ttl=64 time=0.063 ms 64 bytes from cc-3dfr.bcrc.local (10.18.62.60): icmp\_seq=2 ttl=64 time=0.071 ms (base) [modeleval@cc-3dfr ~]$ nslookup cc-3dfr Server: 10.18.10.5 Address: 10.18.10.5#53 ## Name: cc-3dfr.bcrc.local Address: 10.18.62.60 Any guidance to this problem would be greatly appreciated! Thank you! Best, Bill McLaughlin and Thomas Parry ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [December 23, 2022, 7:03am UTC](https://community.openpbs.org/t/jobs-stuck-in-queue-need-to-use-root-access-to-qrun-to-initiate-each-job/3396/2 "2022-12-23T07:03:00Z")
Please submit 2 jobs as below and share the output of the below commands ```auto qstat -Bf qmgr -c 'p q batch' qsub -q batch -- /bin/sleep 100 qsub -q batch -- /bin/sleep 100 qstat -answ1 qstat -f < jobid > ; example : qstat -f 101 tracejob < jobid > ; example tracejob 101 ``` If the job is in the queue - check the scheduler logs against this job id ( probably increase the log level , if you do not see much information and submit another job) - check the server logs against t his job id (probably increase log level , if you do not see much information and submit annother job) ---
**Author:** ![alexis.cousein](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/alexis.cousein/32/265_2.png) [@alexis.cousein](https://community.openpbs.org/u/alexis.cousein)\ **Post date:** [December 27, 2022, 8:14am UTC](https://community.openpbs.org/t/jobs-stuck-in-queue-need-to-use-root-access-to-qrun-to-initiate-each-job/3396/3 "2022-12-27T08:14:08Z")
If qrun without specifying nodes runs the job, then a special scheduler iteration runs the job. That lets the scheduler ignore the state of the queue (is it started?) and any limits. As someone else said: crank up the log events and look at the scheduler logs, and also look at the the job comment which is normally set by the scheduler when it considers a job but does not run it. If there is none the scheduler hasn’t considered it…perhaps the server “scheduling” attribute is set to false. ---
**Author:** ![wamclaughlin](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/wamclaughlin/32/619_2.png) [@wamclaughlin](https://community.openpbs.org/u/wamclaughlin)\ **Post date:** [January 3, 2023, 4:41pm UTC](https://community.openpbs.org/t/jobs-stuck-in-queue-need-to-use-root-access-to-qrun-to-initiate-each-job/3396/4 "2023-01-03T16:41:22Z")
Thank you for taking the time to reply! Here are the outputs for your requested commands. **1.** > [@adarsh](#): > > `qstat -Bf` > (base) [modeleval@cc-3dfr ~]$ qstat -Bf > Server: cc-3dfr > server\_state = Active > server\_host = cc-3dfr.bcrc.local > scheduling = True > total\_jobs = 0 > state\_count = Transit:0 Queued:0 Held:0 Waiting:0 Running:0 Exiting:0 Begun:0 > log\_events = 511 > mailer = /usr/sbin/sendmail > mail\_from = adm > query\_other\_jobs = True > resources\_default.ncpus = 1 > default\_chunk.ncpus = 1 > scheduler\_iteration = 600 > flatuid = True > resv\_enable = True > node\_fail\_requeue = 310 > max\_array\_size = 10000 > pbs\_license\_min = 0 > pbs\_license\_max = 2147483647 > pbs\_license\_linger\_time = 31536000 > license\_count = Avail\_Global:1000000 Avail\_Local:1000000 Used:0 High\_Use:0 > pbs\_version = 20.0.0 > eligible\_time\_enable = False > max\_concurrent\_provision = 5 > max\_job\_sequence\_id = 9999999 **2.** > [@adarsh](#): > > `qmgr -c 'p q batch'` > (base) [modeleval@cc-3dfr ~]$ qmgr -c ‘p q batch’ > Create queues and set their attributes. > Create and define queue batch > create queue batch > set queue batch queue\_type = Execution > set queue batch resources\_default.ncpus = 1 > set queue batch resources\_default.nodect = 1 > set queue batch resources\_default.nodes = 1 > set queue batch resources\_default.walltime = 01:00:00 > set queue batch enabled = True > set queue batch started = True **3.** > [@adarsh](#): > > `qsub -q batch -- /bin/sleep 100` > (base) [modeleval@cc-3dfr ~]$ qsub -q batch – /bin/sleep 100 > 2537.cc-3dfr **4.** > [@adarsh](#): > > `qsub -q batch -- /bin/sleep 100` > (base) [modeleval@cc-3dfr ~]$ qsub -q batch – /bin/sleep 100 > 2538.cc-3dfr **5.** > [@adarsh](#): > > `qstat -answ1` > (base) [modeleval@cc-3dfr ~]$ qstat -answ1 > cc-3dfr: > Req’d Req’d Elap > Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time > > * * * > > 2537.cc-3dfr modeleval batch STDIN – 1 1 – 01:00 Q – – > – > 2538.cc-3dfr modeleval batch STDIN – 1 1 – 01:00 Q – – > – **6.** > [@adarsh](#): > > `qstat -f < jobid > ; example : qstat -f 101` > (base) [modeleval@cc-3dfr ~]$ qstat -f 2537 > Job Id: 2537.cc-3dfr > Job\_Name = STDIN > Job\_Owner = modeleval@cc-3dfr > job\_state = Q > queue = batch > server = cc-3dfr > Checkpoint = u > ctime = Thu Dec 29 15:05:51 2022 > Error\_Path = cc-3dfr:/home/modeleval/STDIN.e2537 > Hold\_Types = n > Join\_Path = n > Keep\_Files = n > Mail\_Points = a > mtime = Thu Dec 29 15:05:51 2022 > Output\_Path = cc-3dfr:/home/modeleval/STDIN.o2537 > Priority = 0 > qtime = Thu Dec 29 15:05:51 2022 > Rerunable = True > Resource\_List.ncpus = 1 > Resource\_List.nodect = 1 > Resource\_List.nodes = 1 > Resource\_List.place = scatter > Resource\_List.select = 1:ncpus=1 > Resource\_List.walltime = 01:00:00 > substate = 10 > Variable\_List = PBS\_O\_HOME=/home/modeleval,PBS\_O\_LANG=en\_US.UTF-8, > PBS\_O\_LOGNAME=modeleval, > PBS\_O\_PATH=/home/modeleval/anaconda3/bin:/home/modeleval/anaconda3/con > dabin:/home/modeleval/google-cloud-sdk/bin:/usr/local/bin:/usr/bin:/usr > /local/sbin:/usr/sbin:/opt/pbs/bin:/home/modeleval/ResiRole/FEATURE/fea > ture-3.0.0/src:/home/modeleval/ResiRole/FEATURE/feature-3.0.0/tools/bin > :/home/modeleval/bin,PBS\_O\_MAIL=/var/spool/mail/modeleval, > PBS\_O\_SHELL=/bin/bash,PBS\_O\_WORKDIR=/home/modeleval,PBS\_O\_SYSTEM=Linux, > PBS\_O\_QUEUE=batch,PBS\_O\_HOST=cc-3dfr > etime = Thu Dec 29 15:05:51 2022 > Submit\_arguments = -q batch – /bin/sleep 100 > executable = jsdl-hpcpa:Executable\>/bin/sleep\ > argument\_list = jsdl-hpcpa:Argument\>100\ > project = \_pbs\_project\_default > Submit\_Host = cc-3dfr **7.** > [@adarsh](#): > > `tracejob < jobid > ; example tracejob 101` > (base) [modeleval@cc-3dfr ~]$ tracejob 2537 > Job: 2537.cc-3dfr > 12/29/2022 15:05:51 S enqueuing into batch, state Q hop 1 > 12/29/2022 15:05:51 S Job Queued at request of modeleval@cc-3dfr, owner = modeleval@cc-3dfr, job name = STDIN, queue = batch Additionally, here are the logs against the submitted job ID’s (2537 and 2538): **Scheduler\_log:** > 12/29/2022 14:59:12;0002;pbs\_sched;Svr;Log;Log opened > 12/29/2022 14:59:12;0002;pbs\_sched;Svr;pbs\_sched;pbs\_version=20.0.0 > 12/29/2022 14:59:12;0002;pbs\_sched;Svr;pbs\_sched;pbs\_build=mach=N/A:security=N/A:configure\_args=N/A > 12/29/2022 14:59:12;0002;pbs\_sched;Svr;pbs\_sched;hostname=cc-3dfr;pbs\_leaf\_name=N/A;pbs\_mom\_node\_name=N/A > 12/29/2022 14:59:12;0002;pbs\_sched;Svr;pbs\_sched;ipv4 interface lo: cc-3dfr.bcrc.local > 12/29/2022 14:59:12;0002;pbs\_sched;Svr;pbs\_sched;ipv4 interface eno1: cc-3dfr.bcrc.local > 12/29/2022 14:59:12;0002;pbs\_sched;Svr;pbs\_sched;ipv4 interface virbr0: cc-3dfr > 12/29/2022 14:59:12;0002;pbs\_sched;Svr;pbs\_sched;ipv6 interface lo: cc-3dfr.bcrc.local > 12/29/2022 14:59:12;0002;pbs\_sched;Svr;pbs\_sched;ipv6 interface eno1: cc-3dfr > 12/29/2022 14:59:12;0002;pbs\_sched;n/a;setup\_env;read environment from /var/spool/pbs/pbs\_environment > 12/29/2022 14:59:12;0006;pbs\_sched;Fil;pbs\_sched;Version 20.0.0, started, initialization type = 0 > 12/29/2022 14:59:12;0002;pbs\_sched;Svr;sched\_main;/opt/pbs/sbin/pbs\_sched startup pid 438795 > 12/29/2022 14:59:12;0040;pbs\_sched;Fil;fairshare usage;Creating usage database for fairshare > 12/29/2022 14:59:12;0080;pbs\_sched;Req;;Launching 12 worker threads > 12/29/2022 14:59:16;0001;pbs\_sched;Svr;pbs\_sched;Access from host not allowed, or unknown host (15008) in open\_server\_conns, Couldn’t register the scheduler default with connected server **Server\_Log** > 12/29/202215:05:51;0100;Server@cc3dfr;Job;2537.cc3dfr;enqueuing mailto: 15:05:51;0100;Server@cc-3dfr;Job;2537.cc-3dfr;enqueuing into batch, state Q hop 1 > 12/29/2022 15:05:51;0008;Server@cc-3dfr;Job;2537.cc-3dfr;Job mailto: 15:05:51;0008;Server@cc-3dfr;Job;2537.cc-3dfr;Job Queued at request of modeleval@cc-3dfr, owner = modeleval@cc-3dfr, job name = STDIN, queue = batch > 12/29/2022 15:05:53;0100;Server@cc-3dfr;Req;;Type 0 request received from root@cc-3dfr, sock=19 > 12/29/2022 15:05:53;0100;Server@cc-3dfr;Req;;Type 95 request received from root@cc-3dfr, sock=20 > 12/29/2022 15:05:53;0100;Server@cc-3dfr;Req;;Type 98 request received from root@cc-3dfr, sock=19 > 12/29/2022 15:05:53;00a0;Server@cc-3dfr;Req;req\_reject;Reject reply code=15008, aux=0, type=98, from root@cc-3dfr > 12/29/2022 15:05:55;0100;Server@cc-3dfr;Req;;Type 0 request received from root@cc-3dfr, sock=19 > 12/29/2022 15:05:55;0100;Server@cc-3dfr;Req;;Type 1 request received from modeleval@cc-3dfr, sock=17 > 12/29/2022 15:05:55;0100;Server@cc-3dfr;Job;2538.cc-3dfr;enqueuing(mailto: 15:05:55;0100;Server@cc-3dfr;Job;2538.cc-3dfr;enqueuing) into batch, state Q hop 1 > 12/29/2022 [15:05:55;0008;Server@cc-3dfr;Job;2538.cc-3dfr;Job](mailto: 15:05:55;0008;Server@cc-3dfr;Job;2538.cc-3dfr;Job) Queued at request of modeleval@cc-3dfr, owner = modeleval@cc-3dfr, job name = STDIN, queue = batch > 12/29/2022 15:05:55;0100;Server@cc-3dfr;Req;;Type 95 request received from root@cc-3dfr, sock=20 > 12/29/2022 15:05:55;0100;Server@cc-3dfr;Req;;Type 98 request received from root@cc-3dfr, sock=19 > 12/29/2022 15:05:55;00a0;Server@cc-3dfr;Req;req\_reject;Reject reply code=15008, aux=0, type=98, from root@cc-3dfr > 12/29/2022 15:05:57;0100;Server@cc-3dfr;Req;;Type 0 request received from root@cc-3dfr, sock=19 > 12/29/2022 15:05:57;0100;Server@cc-3dfr;Req;;Type 95 request received from root@cc-3dfr, sock=20 > 12/29/2022 15:05:57;0100;Server@cc-3dfr;Req;;Type 98 request received from root@cc-3dfr, sock=19 Sincerely, Bill and Thomas ---
**Author:** ![wamclaughlin](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/wamclaughlin/32/619_2.png) [@wamclaughlin](https://community.openpbs.org/u/wamclaughlin)\ **Post date:** [January 3, 2023, 4:45pm UTC](https://community.openpbs.org/t/jobs-stuck-in-queue-need-to-use-root-access-to-qrun-to-initiate-each-job/3396/5 "2023-01-03T16:45:59Z")
Thank you for the suggestion, Alexis. FYI we ran the following command. > qmgr -c “set server scheduling=true” It appears that subsequently the jobs remain stuck in the queue. Also, when we force a job to run from the queue, this is the command we use as root user: > qrun -H cc-3dfr \< jobid \> Is there another attribute that we possible need to modify? Sincerely, Bill and Thomas ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [January 4, 2023, 3:16am UTC](https://community.openpbs.org/t/jobs-stuck-in-queue-need-to-use-root-access-to-qrun-to-initiate-each-job/3396/6 "2023-01-04T03:16:06Z")
> [@wamclaughlin](#): > > 12/29/2022 14:59:16;0001;pbs\_sched;Svr;pbs\_sched;Access from host not allowed, or unknown host (15008) in open\_server\_conns, Couldn’t register the scheduler default with connected server Please check the below 1. Make sure ports 15001 to 15009 are opened for communication 2. firewall are not blocking the ports and SELinux is disabled and system is rebooted if you disable it now. 3. [Qstat : invalid credential =\> due to name resolution - #4 by adarsh](https://community.openpbs.org/t/qstat-invalid-credential-due-to-name-resolution/3400/4) [use the short and fqdn in the clientsfile and check whether it does the trick] ---
**Author:** ![wamclaughlin](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/wamclaughlin/32/619_2.png) [@wamclaughlin](https://community.openpbs.org/u/wamclaughlin)\ **Post date:** [January 11, 2023, 6:44pm UTC](https://community.openpbs.org/t/jobs-stuck-in-queue-need-to-use-root-access-to-qrun-to-initiate-each-job/3396/7 "2023-01-11T18:44:51Z")
Dear Adarsh, Thank you for your comments. Please know that we had to perform an upgrade of the system and in the process I had to reinstall OpenPBS and PostgreSQL. Please see a new thread entitled “Difficulty with installation”; and please possibly provide advice on how to proceed from there. Sincerely, Bill McLaughlin