# Schedulers doesn't seem to be holding jobs **URL:** \ **Category:** Users/Site Administrators\ **Created:** [June 15, 2019, 12:46am UTC](https://community.openpbs.org/t/schedulers-doesnt-seem-to-be-holding-jobs/1654 "2019-06-15T00:46:21Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![brownwrap](https://avatars.discourse-cdn.com/v4/letter/b/94ad74/32.png) [@brownwrap](https://community.openpbs.org/u/brownwrap)\ **Post date:** [June 15, 2019, 12:46am UTC](https://community.openpbs.org/t/schedulers-doesnt-seem-to-be-holding-jobs/1654/1 "2019-06-15T00:46:21Z")
I have a cluster, 28 nodes, 40 cores per node for a total of 1120 cores. User 1 submit a job that asks for 1000 cores user 2 submits a job that ask for 400 cores while User 1’s job is running User 2 doesn’t get queued up, instead it tries to run and aborts. Am I missing a setting or should PBS queue up the job until resources are available? Thanks. ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [June 15, 2019, 7:27am UTC](https://community.openpbs.org/t/schedulers-doesnt-seem-to-be-holding-jobs/1654/2 "2019-06-15T07:27:38Z")
Are you sure that HT is not enabled on these nodes ? Please share the below: 1. job script or qsub statement used by User1 and User2 2. `qstat -fx ` and `qstat -fx ` 3. `pbsnodes -aSj` command output 4. Submit User1 job and then when it is running, run the below scripts and share the output pbsnodes -av | grep resources\_available.ncpus | cut -d’=’ -f2|awk ‘{ sum+=$1} END {print sum}’ pbsnodes -av | grep resources\_assigned.ncpus | cut -d’=’ -f2|awk ‘{ sum+=$1} END {print sum}’ ---
**Author:** ![brownwrap](https://avatars.discourse-cdn.com/v4/letter/b/94ad74/32.png) [@brownwrap](https://community.openpbs.org/u/brownwrap)\ **Post date:** [June 15, 2019, 5:16pm UTC](https://community.openpbs.org/t/schedulers-doesnt-seem-to-be-holding-jobs/1654/3 "2019-06-15T17:16:16Z")
Neither of the users is around today, so I can’t get them to submit their jobs, but I noticed something in the user2’s job that looks odd: #PBS -l select=20:ncpus=40:mpiprocs=40,walltime=1:00:00 Users runs jobs at anytime just fine. I don’t know where his job submission script is. User2 can run fine if no one else is running. But if user2 submit his job while others are running it immediately aborts. User1 is using 1000 of the 1120 cores, so User2’s job should be held, but I don’t think he has constructed his submission script correctly. ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [June 15, 2019, 9:52pm UTC](https://community.openpbs.org/t/schedulers-doesnt-seem-to-be-holding-jobs/1654/4 "2019-06-15T21:52:10Z")
Thank you please share the information whenever possible. > [@brownwrap](#): > > #PBS -l select=20:ncpus=40:mpiprocs=40,walltime=1:00:00 The job request is correct, user2 is requesting 800 cores with a job walltime of 1 hour. Could you please share the PBS Pro OSS version you are running ? ( e.g., qstat --version ) ---
**Author:** ![brownwrap](https://avatars.discourse-cdn.com/v4/letter/b/94ad74/32.png) [@brownwrap](https://community.openpbs.org/u/brownwrap)\ **Post date:** [June 15, 2019, 10:37pm UTC](https://community.openpbs.org/t/schedulers-doesnt-seem-to-be-holding-jobs/1654/5 "2019-06-15T22:37:25Z")
Here is the version: [ramos@sandy1 ~] qstat --version pbs\_version = 14.1.2 [ramos@sandy1 ~] I was thinking the line should be: #PBS -l select=20:ppn=40,walltime=1:00:00 ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [June 16, 2019, 6:46am UTC](https://community.openpbs.org/t/schedulers-doesnt-seem-to-be-holding-jobs/1654/6 "2019-06-16T06:46:41Z")
Thank you > [@brownwrap](#): > > #PBS -l select=20:ppn=40,walltime=1:00:00 ppn=40 is a old syntax was used in PBS Pro version 9 and before . select and ncpus are correct here. PBS converts old-style resource requests to select and place statements. See the 19.2.1 UG, section 4.8.3, “Conversion of Old Style to New”, on page UG-72 > **[High-performance Computing (HPC) and Cloud Solutions | Altair](https://altair.com/hpc-cloud-applications)** > > Altair’s industry leading HPC tools, solutions & applications let you orchestrate, visualize, optimize, and analyse your most demanding workloads, easily migrating to the cloud and eliminating I/O bottlenecks. Explore more. ---
**Author:** ![brownwrap](https://avatars.discourse-cdn.com/v4/letter/b/94ad74/32.png) [@brownwrap](https://community.openpbs.org/u/brownwrap)\ **Post date:** [June 16, 2019, 6:04pm UTC](https://community.openpbs.org/t/schedulers-doesnt-seem-to-be-holding-jobs/1654/7 "2019-06-16T18:04:18Z")
> [@adarsh](#): > > pbsnodes -av | grep resources\_assigned.ncpus | cut -d’=’ -f2|awk ‘{ sum+=$1} END {print sum}’ Thank you. I want to rule out a format issue. Still the weekend, so no users are in, but here is a job that immediately fails: more job-15170 06/14/2019 15:26:34 L Considering job to run 06/14/2019 15:26:34 S Job Queued at request of viner@sandy1.local, owner = viner@sandy1.local, job name = ColdForecast.20, queue = workq 06/14/2019 15:26:34 S Job Run at request of Scheduler@smaster1 on exec\_vnode (compute-0-13:ncpus=40)+(compute-0-14:ncpus=40)+(compute-0-15:ncpus=40)+(compute-0-16:ncpus=40)+(compute-0-17:ncpus=40)+(compute-0-8:ncpus=40)+(compute-0-18:ncpus= 40)+(compute-0-19:ncpus=40)+(compute-0-20:ncpus=40)+(compute-0-21:ncpus=40) 06/14/2019 15:26:34 S Job Modified at request of Scheduler@smaster1 06/14/2019 15:26:34 L Job run 06/14/2019 15:26:34 S enqueuing into workq, state 1 hop 1 06/14/2019 15:26:34 S Obit received momhop:1 serverhop:1 state:4 substate:42 06/14/2019 15:26:34 A queue=workq 06/14/2019 15:26:34 A user=viner group=viner account=“None” project=\_pbs\_project\_default jobname=ColdForecast.20 queue=workq ctime=1560551194 qtime=1560551194 etime=1560551194 start=1560551194 exec\_host=compute-0-13/0_40+compute-0-14/0_40+compute-0-15/0_40+compute-0-16/0_40+compute-0-17/0_40+compute-0-8/0_40+compute-0-18/0_40+compute-0-19/0_40+compute-0- 20/0_40+compute-0-21/0_40 exec\_vnode=(compute-0-13:ncpus=40)+(compute-0-14:ncpus=40)+(compute-0-15:ncpus=40)+(compute-0-16:ncpus=40)+(compute-0-17:ncpus=40)+(compute-0-8:ncpus=40)+(compute- 0-18:ncpus=40)+(compute-0-19:ncpus=40)+(compute-0-20:ncpus=40)+(compute-0-21:ncpus=40) Resource\_List.mpiprocs=400 Resource\_List.ncpus=400 Resource\_List.nodect=10 Resource\_List.place=free Resource\_List.select=10:ncpus=40:mpiprocs=40 Resource\_List.walltime=01:00:00 resource\_assigned.ncpus=400 06/14/2019 15:26:35 S Exit\_status=1 resources\_used.cpupercent=0 resources\_used.cput=00:00:00 resources\_used.mem=412kb resources\_used.ncpus=400 resources\_used.vmem=12864kb resources\_used.walltime=00:00:00 06/14/2019 15:26:35 A user=viner group=viner account=“None” project=\_pbs\_project\_default jobname=ColdForecast.20 queue=workq ctime=1560551194 qtime=1560551194 etime=1560551194 start=1560551194 exec\_host=compute-0-13/0_40+compute-0-14/0_40+compute-0-15/0_40+compute-0-16/0_40+compute-0-17/0_40+compute-0-8/0_40+compute-0-18/0_40+compute-0-19/0_40+compute-0- 20/0_40+compute-0-21/0_40 exec\_vnode=(compute-0-13:ncpus=40)+(compute-0-14:ncpus=40)+(compute-0-15:ncpus=40)+(compute-0-16:ncpus=40)+(compute-0-17:ncpus=40)+(compute-0-8:ncpus=40)+(compute- 0-18:ncpus=40)+(compute-0-19:ncpus=40)+(compute-0-20:ncpus=40)+(compute-0-21:ncpus=40) Resource\_List.mpiprocs=400 Resource\_List.ncpus=400 Resource\_List.nodect=10 Resource\_List.place=free Resource\_List.select=10:ncpus=40:mpiprocs=40 Resource\_List.walltime=01:00:00 session=121410 end=1560551195 Exit\_status=1 resources\_used.cpupercent=0 resources\_used.cput=00:00:00 resources\_used.mem=412kb resources\_used.ncpus=400 resources\_used.vmem=12864kb resources\_used.walltime=00:00:00 run\_count=1 [root@smaster1 tmp]# ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [June 16, 2019, 7:48pm UTC](https://community.openpbs.org/t/schedulers-doesnt-seem-to-be-holding-jobs/1654/8 "2019-06-16T19:48:30Z")
> [@brownwrap](#): > > 06/14/2019 15:26:35 S Exit\_status=1 resources\_used.cpupercent=0 resources\_used.cput=00:00:00 resources\_used.mem=412kb resources\_used.ncpus=400 resources\_used.vmem=12864kb > resources\_used.walltime=00:00:00 Please note that the exit status is 1 ( which means the application batch command that the user intended to run via the PBS Script failed ) , hence job has exited . In this scenario, could you please test whether you can run the user script without using PBS Pro on the compute nodes and check whether it executes successfully. Thank you for sharing the information. You can print the script the user has submitted by running the below command as a root user `printjob -s ` ---
**Author:** ![brownwrap](https://avatars.discourse-cdn.com/v4/letter/b/94ad74/32.png) [@brownwrap](https://community.openpbs.org/u/brownwrap)\ **Post date:** [June 16, 2019, 9:33pm UTC](https://community.openpbs.org/t/schedulers-doesnt-seem-to-be-holding-jobs/1654/9 "2019-06-16T21:33:01Z")
The script will run, if it is the only job running. that is the root of the problem. The script will run by itself, but not if the other user is running his. Here is the mom log of one of the compute nodes: 06/14/2019 15:26:34;0008;pbs\_mom;Job;15170.smaster1;nprocs: 488, cantstat: 0, nomem: 0, skipped: 0, cached: 0, max excluded PID: 0 06/14/2019 15:26:34;0008;pbs\_mom;Job;15170.smaster1;Started, pid = 121410 06/14/2019 15:26:34;0080;pbs\_mom;Job;15170.smaster1;task 00000001 terminated 06/14/2019 15:26:34;0008;pbs\_mom;Job;15170.smaster1;Terminated 06/14/2019 15:26:34;0100;pbs\_mom;Job;15170.smaster1;task 00000001 cput= 0:00:00 06/14/2019 15:26:34;0008;pbs\_mom;Job;15170.smaster1;kill\_job 06/14/2019 15:26:34;0100;pbs\_mom;Job;15170.smaster1;compute-0-13 cput= 0:00:00 mem=412kb 06/14/2019 15:26:34;0100;pbs\_mom;Job;15170.smaster1;compute-0-14.local cput= 0:00:00 mem=0kb 06/14/2019 15:26:34;0100;pbs\_mom;Job;15170.smaster1;compute-0-15.local cput= 0:00:00 mem=0kb 06/14/2019 15:26:34;0100;pbs\_mom;Job;15170.smaster1;compute-0-16.local cput= 0:00:00 mem=0kb 06/14/2019 15:26:34;0100;pbs\_mom;Job;15170.smaster1;compute-0-17.local cput= 0:00:00 mem=0kb 06/14/2019 15:26:34;0100;pbs\_mom;Job;15170.smaster1;compute-0-8.local cput= 0:00:00 mem=0kb 06/14/2019 15:26:34;0100;pbs\_mom;Job;15170.smaster1;compute-0-18.local cput= 0:00:00 mem=0kb 06/14/2019 15:26:34;0100;pbs\_mom;Job;15170.smaster1;compute-0-19.local cput= 0:00:00 mem=0kb 06/14/2019 15:26:34;0100;pbs\_mom;Job;15170.smaster1;compute-0-20.local cput= 0:00:00 mem=0kb 06/14/2019 15:26:34;0100;pbs\_mom;Job;15170.smaster1;compute-0-21.local cput= 0:00:00 mem=0kb 06/14/2019 15:26:34;0008;pbs\_mom;Job;15170.smaster1;no active tasks 06/14/2019 15:26:34;0100;pbs\_mom;Job;15170.smaster1;Obit sent ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [June 17, 2019, 5:49am UTC](https://community.openpbs.org/t/schedulers-doesnt-seem-to-be-holding-jobs/1654/10 "2019-06-17T05:49:05Z")
Thank you for the mom logs, it does not say anything as job is not using any cput or walltime and is immediately exiting with Exit\_Status = 1 Could you please submit the below and share the output: ``` qsub -l select=20:ncpus=40 -- /bin/sleep 10000 qsub -l select=20:ncpus=40 -- /bin/sleep 10000 qsub -l select=20:ncpus=40 -- /bin/sleep 10000 qstat -answ1 pbsnode -aSj ``` If this fails, then you have to share us pbs\_diag output by running `$PBS_EXEC/unsupported/pbs_diag -j ` , the output of this command is a tar.gz file stored in the /root/ folder. ---
**Author:** ![brownwrap](https://avatars.discourse-cdn.com/v4/letter/b/94ad74/32.png) [@brownwrap](https://community.openpbs.org/u/brownwrap)\ **Post date:** [June 17, 2019, 1:47pm UTC](https://community.openpbs.org/t/schedulers-doesnt-seem-to-be-holding-jobs/1654/11 "2019-06-17T13:47:24Z")
This ran as expected: 15223.smaster1 STDIN ramos 00:00:00 R workq 15224.smaster1 STDIN ramos 0 Q workq 15225.smaster1 STDIN ramos 0 Q workq ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [June 18, 2019, 7:21am UTC](https://community.openpbs.org/t/schedulers-doesnt-seem-to-be-holding-jobs/1654/12 "2019-06-18T07:21:55Z")
Thank you. There are no issues with the cluster configuration. It is to do with the job script that user is submitting, which is failing instantly. If you can share the job script (removing any classifieds) that is failing immediately we can check.