Hi,
I am bit new to PBS, Need some help in this.
I require to reserve some resources within a normal queue by a user to run priority job at some scheduled time. Also we have node binding configuration in pbs, can some one help how to do it, i mean what changes (in job submission/configuration ) required step by step.
PBS documents says, Advance / standing reservation can be done if nodes are not binded with queues, i.e nodes should not be assigned to any queue.
Thanks
Varun
Node to queue mapping is not recommended and is not efficient in terms of resource usage by the end users. It is recommended to configure Qlists instead.
You can use reservations ( Advance or Standing) here with Qlist configuration without any issues.
Thanks for the reply.
yes, vnodes have been configured with qlist configuration, also we have some vnodes associated with multiple queues.
snippet of one of the vnode :
Node node_970_0
Mom = nid00077,nid00845,nid01553,nid01229,nid01997,nid00461,nid00781
Port = 15002
pbs_version = PBSPro_13.0.409.180551
ntype = PBS
state = job-exclusive
pcpus = 36
resources_available.arch = XT
resources_available.host = node_970
resources_available.mem = 67108864kb
resources_available.ncpus = 36
resources_available.nppus = 18
resources_available.PBShost = node
resources_available.PBSnid = 970
resources_available.PBSorder = 920
resources_available.PBSseg = 0
resources_available.Qlist = queue1,queue2
resources_available.vnode = node_970_0
resources_available.vntype = compute
resources_available.vps_per_ppu = 2
resources_assigned.accelerator_memory = 0kb
resources_assigned.hbmem = 0kb
resources_assigned.mem = 0kb
resources_assigned.naccelerators = 0
resources_assigned.ncpus = 36
resources_assigned.netwins = 0
resources_assigned.nppus = 0
resources_assigned.vmem = 0kb
resv_enable = True
sharing = force_exclhost
in_multivnode_host = 1
license = l
Now, How can we use reservations with Qlist ?
I want to reserve vnodes only from queue1 to assign a job at some specific time.
Thank you . Please share the output of this command qmgr -c “print queue queue1”
Here is the output:
create queue queue1
set queue queue1 queue_type = Execution
set queue queue1 max_queued = [o:PBS_ALL=200]
set queue queue1 acl_user_enable = TRUE
set queue queue1 acl_users = varun
set queue queue1 acl_users += test_usr
set queue queue1 resources_max.ncpus = 21600
set queue queue1 resources_max.nodect = 600
set queue queue1 resources_max.walltime = 8:00:00
set queue queue1 resources_default.vntype = compute
set queue queue1 resources_default.walltime = 8:00:00
set queue queue1 acl_group_enable = TRUE
set queue queue1 acl_groups = staff
set queue queue1 default_chunk.Qlist = queue1
set queue queue1 default_chunk.vntype = compute
set queue queue1 max_run_res.ncpus = [o:PBS_ALL=21600]
set queue queue1 enabled = TRUE
set queue queue1 started = TRUE
Thank you , please run the below line for advance reservation:
pbs_rsub -R 201907211520 -E 201908211800 -N PBSRESERVATION -U root,pbsdata,pbsworksuser -l select=2:ncpus=36:Qlist=queue1:vntype=compute -l place=scatter:excl
Thanks for the reply.
I have tried the above procedure, which successfully created a new reservation queue reserving nodes from queue “queue1”.
Now the thing is, Advance reservation creates a new name queue (ex: R12345) which should be mentioned in job script (PBS -q R12345) while submitting the job.
Moving further…
Can we customize “reservation queue-name” (like R12345 --> My_Resv_Q) ?
Is there any option to use reserved nodes from reservation queue(R12345) as normal, when no priority job is running.
Suppose a job is running in Reserved Time slot in reserved queue, let say JobA running in R12345. If we require to submit a more priority JobB in R12345, what will happen ? how can we acheive JObB to run immediately,if we can afford JObA to hold.
Please suggest the possibilities.
**** Just enjoying playing with PBS & exploring the new possibilities ****
No, the queue name is created dynamically based on reservation confirmation
Short answer is NO, when the reservation is ACTIVE.
If you see when the reservation is confirmed, the nodes are kind of mapped to the Reservation queue.
You can check that by running pbsnodes -av | grep -i resv , you will see something like below
resv = R11257.servername , these are the nodes that belong to respective reservation queue.
The reservation is created to run priority jobs with a walltime . The reservation is created to expect resource available by the workload you would have at that time.
Any jobs can be submitted or queue moved to RESERVATION queue. The reservation nodes are available to all the queues before the RESERVATION start-time. Please note walltime plays a crucial role when using reservation and submitting jobs to other queues before the reservation start time.
The basics of reservation is to reserve compute resources for a specified amount of time (guarantied) for a specific priority workload ( jobs ) .
There might be many ways:
Please check 4.9.21.12 Examples of Using Resource Permissions in Job Sorting
Formula in the PBS Professional 19.2 Administrator’s Guide. Page: AG-155
Thank you
Many Thanks to you Adarsh, This was really helpful.
Will explore more options from the Admin guide.
Thankyou
varun
If you build from source, look at the code bracketed by comments with “localmod 075”. This is a NAS localmod that allows you to assign names to reservations that can then be used when submitting jobs.
Ex.
15:33 pbspl1 ~.$ pbs_rsub -N my_name -D 100 -l select=1:model=san -R 010101
R6799955.pbspl1.nas.nasa.gov UNCONFIRMED
15:34 pbspl1 ~.$ pbs_rstat | grep -e Resv -e my_name
Resv ID Queue User State Start / Duration / End
my_name R6799955 dtalcott CO Thu Aug 01 01:01 / 100 / Thu Aug 01 01:02
15:35 pbspl1 ~.$ echo sleep 10 | qsub -q my_name -l select=1:model=san:aoe=sles12 -o foo.out -joe
6799995.pbspl1.nas.nasa.gov
15:35 pbspl1 ~.$ qstat -u dtalcott
Req'd Elap Rem
SeqNo User Queue Jobname CPUs Nds wallt Ss wallt Eff wallt
------- -------- -------- ------- ---- --- ----- -- ----- --- -----
6799995 dtalcott R6799955 STDIN 1 1 00:02 Q 00:00 -- 00:02
There is a limitation to this code: multiple reservations can have the same name. It is unspecified which reservation will be picked. In practice here, this has not been a problem and is sometimes a feature.
Hi,
I have tried what you shared, and below is the output while creating reservation:
R4536261.s R4536261 varun@el CO Today 15:00 / 300 / Today 15:05
Expected “Resv ID” is not created with name “my_res”
Please share the output of pbs_rstat -f
varun@elogin04:~> pbs_rsub -R 1510 -E 1515 -N my_res -l select=2:ncpus=36:Qlist=queue1:vntype=compute -l place=scatter:excl
R4550559.sdb UNCONFIRMED
varun@elogin04:~>
varun@elogin04:~> pbs_rstat
Resv ID Queue User State Start / Duration / End
---------------------------------------------------------------------
R4550559.s R4550559 varun@el UN Today 15:10 / 300 / Today 15:15
varun@elogin04:~> pbs_rstat -f
Resv ID: R4550559.sdb
Reserve_Name = my_res
Reserve_Owner = varun@elogin04
reserve_state = RESV_CONFIRMED
reserve_substate = 2
reserve_start = Wed Jul 31 15:10:00 2019
reserve_end = Wed Jul 31 15:15:00 2019
reserve_duration = 300
queue = R4550559
Resource_List.ncpus = 72
Resource_List.nodect = 2
Resource_List.select = 2:ncpus=36:Qlist=queue1:vntype=compute
Resource_List.place = scatter:excl
Resource_List.walltime = 00:05:00
resv_nodes = (node_6_0:ncpus=36)+(node_6_1)+(node_8_0:ncpus=36)+(node_8_1)
Authorized_Users = varun@elogin04
server = sdb
ctime = Wed Jul 31 15:07:47 2019
mtime = Wed Jul 31 15:07:47 2019
Variable_List = PBS_O_LOGNAME=varun,PBS_O_HOST=elogin04,PBS_O_MAIL=/var/mail/varun
Your reservation name is my_res, this is created.
Reservation ID is not customisable, it is generated by the system upon requesting a reservation.
yeh, i have found the same,
But It was a reply to Dale’s suggested Example above.
There Resv ID can be assigned with names that can then be used when submitting jobs.
Sorry to be so slow to respond. The pbs_rstat -f output shows that the name got recorded. So, the problem is somewhere in the pbs_rstat code that displays things. I recall making some changes in our part of that code for pbspro 19. You can find these changes (and others) via:
git clone https://github.com/drtoss/pbspro.git drtoss19
cd drtoss19
git checkout nas_mods
Then compare that pbs_rstat.c to yours.