# How to Restrict maximum 2 jobs/per user in each scheduling cycle **URL:** \ **Category:** Users/Site Administrators\ **Created:** [December 17, 2024, 3:33pm UTC](https://community.openpbs.org/t/how-to-restrict-maximum-2-jobs-per-user-in-each-scheduling-cycle/4046 "2024-12-17T15:33:04Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![sahuhem](https://avatars.discourse-cdn.com/v4/letter/s/ecb155/32.png) [@sahuhem](https://community.openpbs.org/u/sahuhem)\ **Post date:** [December 17, 2024, 3:33pm UTC](https://community.openpbs.org/t/how-to-restrict-maximum-2-jobs-per-user-in-each-scheduling-cycle/4046/1 "2024-12-17T15:33:04Z")
Let’s say a user submits 5 Jobs back-to-back and My PBS scheduling cycle is 30 sec. Now All 5 jobs are in Q state . How to restrict only top 2 queued jobs from that user in priority order are evaluated for sending them to Run state (Assuming resources are available for all 5 queued jobs in next scheduling cycle) . ---
**Author:** ![dtalcott](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/dtalcott/32/410_2.png) [@dtalcott](https://community.openpbs.org/u/dtalcott)\ **Post date:** [December 18, 2024, 12:39am UTC](https://community.openpbs.org/t/how-to-restrict-maximum-2-jobs-per-user-in-each-scheduling-cycle/4046/2 "2024-12-18T00:39:41Z")
Have you looked at the max\_run parameter? You could set a limit of 2 jobs running per user with `qmgr -c 'set server max_run="[u:PBS_GENERIC=2]"'` If that is not what you want, what is the higher-level goal? ---
**Author:** ![sahuhem](https://avatars.discourse-cdn.com/v4/letter/s/ecb155/32.png) [@sahuhem](https://community.openpbs.org/u/sahuhem)\ **Post date:** [December 18, 2024, 5:55am UTC](https://community.openpbs.org/t/how-to-restrict-maximum-2-jobs-per-user-in-each-scheduling-cycle/4046/3 "2024-12-18T05:55:02Z")
Hi dtalcott, Thanks for the quick response . However our requirement is bit different We have configured server\_dyn\_res: “VCPU\_AVAIL !/opt/pbs/sbin/free\_vcpus.sh” to limit the number cloud instances .The Scheduler evaluates the FREE\_VCPU dynamic resource in every scheduling cycle(i.e 30 sec) and instead of examining job by job ,scheduler sends all the jobs to Run state those passed through the VCPU evaluation criteria which causes VCPU quota limit . Following Test case scenario shows how to reproduce the issue. Where VCPU = 2 X NCPU (considering HT on) \*Step1: Edited the /opt/pbs/sbin/free\_vcpus.sh to show Avilable free VCPUs to show max capping as 74 \* $ /opt/pbs/sbin/free\_vcpus.sh 74 _Step-2: Submit a 48 core (i.e 96 VCPU) Job .It should wait in Q as expected._ $ qstat -aws|grep -A1 parallel 3079.ip-10-77-162-79 testuser1 parallel CFX – 48 48 – 05:00 Q – Can Never Run: Insufficient amount of server resource: VCPU\_AVAIL (R: 96 A: 76 T: 76) _Step-3: Then Submit 2 consecutive 24 core (i.e 48 VCPU) Jobs . I am expecting 1 should go to Running state and another should wait . But both are going to Running state to initiate instance creation and there by causing over consumption VCPU limit_ $ qstat -aws|grep -A1 parallel 3079.ip-10-77-162-79 testuser1 parallel CFX – 48 48 – 05:00 Q – Can Never Run: Insufficient amount of server resource: VCPU\_AVAIL (R: 96 A: 76 T: 76) 3080.ip-10-77-162-79 testuser1 parallel CFX – 24 24 – 15:00 R – 3081.ip-10-77-162-79 testuser1 parallel CFX – 24 24 – 15:00 R – _2nd Requirement_ : Although I have set backfill\_depth =2 for the queue “parallel” .I am expecting that the scheduler should first reserve the resources top 2 jobs in order before sending lower priority jobs to run state which is not happening in this case (i.e Block all other low priority jobs unless they don’t affect the execution of top 2 jobs). Please note the wall time set is 5 Hrs for first job and other second and third are 15 hours. > > $ qmgr -c "print queue parallel "|grep depth set queue parallel backfill\_depth = 2 > > Let me know if we can tune some of the parameters to meet this requirement. ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [December 18, 2024, 9:07am UTC](https://community.openpbs.org/t/how-to-restrict-maximum-2-jobs-per-user-in-each-scheduling-cycle/4046/4 "2024-12-18T09:07:12Z")
+1 @dtalcott has suggested , there are other limits which can be applied The below might help: - Try using static server-level resources , instead of server\_dyn\_res ( to avoid race condition or runjob hook can be handy), however you will still have jobs starting the same time -5.14.3.2 Static Server-level Resources from [https://help.altair.com/2024.1.0/PBS%20Professional/PBS2024.1.pdf](https://help.altair.com/2024.1.0/PBS%20Professional/PBS2024.1.pdf) - the provisioning hook, if it helps your situation 16.1 Introduction from [https://help.altair.com/2024.1.0/PBS%20Professional/PBS2024.1.pdf](https://help.altair.com/2024.1.0/PBS%20Professional/PBS2024.1.pdf) > [@sahuhem](#): > > (i.e Block all other low priority jobs unless they don’t affect the execution of top 2 jobs) You can use job\_sort\_formula\_threshold , to ignore the jobs that are below the threshold. ![image](https://canada1.discourse-cdn.com/flex030/uploads/pbspro/optimized/1X/d61da98f18ba6a7e6e261c76609a55480ffe0259_2_690x54.png) ---
**Author:** ![sahuhem](https://avatars.discourse-cdn.com/v4/letter/s/ecb155/32.png) [@sahuhem](https://community.openpbs.org/u/sahuhem)\ **Post date:** [December 18, 2024, 2:38pm UTC](https://community.openpbs.org/t/how-to-restrict-maximum-2-jobs-per-user-in-each-scheduling-cycle/4046/5 "2024-12-18T14:38:32Z")
What about the _2nd Requirement:_ backfill\_depth = 2 which is not honored at all. ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [December 18, 2024, 7:54pm UTC](https://community.openpbs.org/t/how-to-restrict-maximum-2-jobs-per-user-in-each-scheduling-cycle/4046/6 "2024-12-18T19:54:07Z")
backfill\_depth is to find the number top jobs that can be filled around the calendared jobs and would not affect the start time of these calendered jobs. The scheduler does not reserve these jobs and its resources, as the situation might vary and in the next scheduling cycle based on the other jobs in the queue, fresh jobs, some other jobs can be a candidate for backfilling. The landscape might change for the backfillable jobs, but the calendared jobs remain intact. backfill\_depth is used to make smaller jobs run between large jobs without affecting the start time of larger jobs, otherwise, smaller jobs gets backfilled and will start to push the large jobs way longer in the future and large jobs would not get time to run. ---
**Author:** ![sahuhem](https://avatars.discourse-cdn.com/v4/letter/s/ecb155/32.png) [@sahuhem](https://community.openpbs.org/u/sahuhem)\ **Post date:** [December 19, 2024, 5:35am UTC](https://community.openpbs.org/t/how-to-restrict-maximum-2-jobs-per-user-in-each-scheduling-cycle/4046/7 "2024-12-19T05:35:55Z")
As per my understanding in order to backfill work properly (i.e to make smaller jobs run between large jobs without affecting the start time of larger jobs) the scheduler should be able to estimate the approximate start time of the job which can be viewed using “qstat -T” command . In my Case the “Est start Time” column showing blank, and the reason could be the scheduler is not aware when approximately the server\_dyn\_res: “VCPU\_AVAIL !/opt/pbs/sbin/free\_vcpus.sh” will be available. So, it is better to disable using backfill\_depth = 0 and let the the jobs run based on priority without backfilling. Please correct me if I am wrong… ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [December 19, 2024, 7:38am UTC](https://community.openpbs.org/t/how-to-restrict-maximum-2-jobs-per-user-in-each-scheduling-cycle/4046/8 "2024-12-19T07:38:14Z")
> [@Created node bucket messages in sched\_logs](https://community.openpbs.org/t/created-node-bucket-messages-in-sched-logs/2740/7): > > I don’t think changing will provisioning\_priority will make a difference. This is just a way to turn off the node bucket node search algorithm. That can only help speed things up. The node bucket algorithm kicks in if you request your nodes with -lplace=excl. I’d revert that change. Do you have strict\_ordering on? Having backfill\_depth=0 will cause your system to idle. If you don’t have strict\_ordering on, it won’t make a difference until jobs start starving (wait time \> 1day). After tha… > [@PBS\_est hook enable fails](https://community.openpbs.org/t/pbs-est-hook-enable-fails/1044): > > Hello, I am attempting to enable the PBS job time estimation hook as documented in the PBS Admin Guide ([https://pbsworks.com/pdfs/PBSAdminGuide14.2.pdf](https://pbsworks.com/pdfs/PBSAdminGuide14.2.pdf)) on page AG-117/118 but I’m receiving an error instead: [~]# qmgr -c ‘set pbshook PBS\_est enabled = true’ qmgr obj=PBS\_est svr=default: PBS\_est does not exist! qmgr: hook error returned from server Is there some pre-requisite to activating this in-built PBS hook? Thanks Siji > [@Qmgr -c 'set server est\_start\_time\_freq ...' fails](https://community.openpbs.org/t/qmgr-c-set-server-est-start-time-freq-fails/218): > > Hi, According to the Admin Guide (v13), the server should be able to compute “estimated start time” for jobs. However, for my installation of v14.1 qmgr complains when I try to set (or even unset) this feature: This command is explicitly snipped from the docs/guide, but it still fails: [root]# qmgr -c ‘set server est\_start\_time\_freq = 3:00:00’ qmgr: Syntax error set server est\_start\_time\_freq = 3:00:00 Even unset’ting it (which would be the default) fails [root@bifrost1 ~]# qmgr -c ‘un… ---
**Author:** ![dtalcott](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/dtalcott/32/410_2.png) [@dtalcott](https://community.openpbs.org/u/dtalcott)\ **Post date:** [December 19, 2024, 6:34pm UTC](https://community.openpbs.org/t/how-to-restrict-maximum-2-jobs-per-user-in-each-scheduling-cycle/4046/9 "2024-12-19T18:34:33Z")
I think the problem with backfill is that it looks at only consumable or node-level resources; that is, things that can be reserved. As adarsh suggested, you might try using a static server level consumable resource whose resources\_available value is adjusted by a periodic hook to match the number of VCPUs defined by the current cloud resources. ---
**Author:** ![alexis.cousein](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/alexis.cousein/32/265_2.png) [@alexis.cousein](https://community.openpbs.org/u/alexis.cousein)\ **Post date:** [December 23, 2024, 4:29pm UTC](https://community.openpbs.org/t/how-to-restrict-maximum-2-jobs-per-user-in-each-scheduling-cycle/4046/10 "2024-12-23T16:29:35Z")
There’s no need to if you want a limit _per scheduling cycle_. Just use a server\_dyn\_res _uncounted_ (no “n” or “q” flag) custom resource and let the script return 2. The scheduler will consume up to 2 each cycle and no more. Obviously the fact it’s “per cycle” means that in practice for many other use cases this is the _wrong_ kind of resource, since it’s easy to have back to back scheduling cycles in which the job in the previous cycle will e.g. not yet have consumed a license, leading to overcommitment. ---
**Author:** ![dtalcott](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/dtalcott/32/410_2.png) [@dtalcott](https://community.openpbs.org/u/dtalcott)\ **Post date:** [December 24, 2024, 4:30pm UTC](https://community.openpbs.org/t/how-to-restrict-maximum-2-jobs-per-user-in-each-scheduling-cycle/4046/11 "2024-12-24T16:30:15Z")
One more thing. You say “My PBS scheduling cycle is 30 sec”. The value of scheduler\_iteration sets the _longest_ interval the server will allow between scheduling cycles. If a new job is queued, the server kicks the scheduler to run right away. Similarly when a job ends. I think this explains your observation where two 48 VCPU jobs started where you expected only one. When the first job was accepted, the server immediately kicked the scheduler. The scheduler sees only the one job and starts it (48 \< 72). Meanwhile, the second job arrives. The server immediately triggers the scheduler again. free\_vcpus.sh again reports 72. That is greater than 48, so the scheduler starts the second job. You could verify this guess by looking at the sched\_logs to see if the two jobs were started in the same scheduling cycle or in two one-after-the-other cycles. ---
**Author:** ![sahuhem](https://avatars.discourse-cdn.com/v4/letter/s/ecb155/32.png) [@sahuhem](https://community.openpbs.org/u/sahuhem)\ **Post date:** [December 30, 2024, 7:26am UTC](https://community.openpbs.org/t/how-to-restrict-maximum-2-jobs-per-user-in-each-scheduling-cycle/4046/12 "2024-12-30T07:26:34Z")
Yes, Your explanation is correct but could you please suggest how to overcome this challenge? ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [December 30, 2024, 11:04am UTC](https://community.openpbs.org/t/how-to-restrict-maximum-2-jobs-per-user-in-each-scheduling-cycle/4046/13 "2024-12-30T11:04:03Z")
Instead of using server\_dyn\_res, try static server-level resource, you can try the below and see whether this would help ```auto * qmgr -c "create resource fvcpus type=long,flag=q" * qmgr -c "set server resources_available.fvcpus=74" # statically set to 74 vcpus * Add fvcpus to the resources: line of the $PBS_HOME/sched_priv/sched_config file ( kill -HUP PID of the pbs_sched> * qsub -l select=1:ncpus=48 -l fvcpus=48 -- /bin/sleep 100 # check qstat -Bf output now ```