How to do PBS users reservations

Users/Site Administrators
1

Cluster size, users and reservations

  • Cluster size 13 nodes
  • Users 3
    o Ashley => guaranteed minimum number of nodes: 4
    o Bob => guaranteed minimum number of nodes: 4
    o Otto

Scenario 1:

  • Job size of all jobs is 3 nodes
  • There are always jobs waiting, but not always from all users
  • Therefore the number of jobs running can vary
    o Ashley: 0 – 3 job (zero only when no job is waiting, as a 3 node job fits in the guaranteed minimum number of nodes)
    o Bob: 0 – 3 job (zero only when no job is waiting, as a 3 node job fits in the guaranteed minimum number of nodes)
    o Otto: 0 - 1 job

how to set such kind of reservation in PBSPro ?

regards
deepak

2

The reservation way:

  • Ashley: pbs_rsub –R 202001741700 –D 13:00 –l select=3:ncpus=1 -l place=scatter:excl
  • Bob: pbs_rsub –R 202001741700 –D 13:00 –l select=3:ncpus=1 -l place=scatter:excl
  • Ashley – to Ashley reservation queue
  • Bob – to Bobs reservation queue
  • Rest of the users will run jobs on the remaining nodes.
  • Also, Ashley / Bob can submit jobs to other queues as well , other than their respective reservation queues

Non-reservation way:
Otherwise some hooks and below configuration:
qmgr -c “create resource ashley type=long,flag=q”
qmgr -c “create resource bob type=long,flag=q”
qmgr -c “create resource totalnodes type=long,flag=q”

Add “ashley,bob,totalnodes,others” to the resources: line in the $PBS_HOME/sched_priv/sched_config and kill -HUP <PID of pbs_sched process>

qmgr -c “set server resources_available.ashley=4”
qmgr -c “set server resources_available.bob=4”
qmgr -c “set server resources_avaialble.totanodes=13”
qmgr -c “set server resources_avaialble.others=5”

You can have server/queue limits to limits number of jobs run by each user ( or control it via server periodic hook based on some logic )

Make sure you have hook that attaches requests accordingly , for example if ashley submits a job then the qsub statement should look below

qsub -l selectXXXXXXXXXXXXXXXXXXXX -l ashley=3 -l totalnodes=3 – /bin/sleep 1000

If any other users (other than ashley , bob) submit the job then

qsub -l selectXXXXXXXXXXXXXXXXXXXX -l totalnodes=3 -l others=3 – /bin/sleep 10000

Hope this is useful or others might have other tricks.

3

Thanks Adarsh for this information

can you also send me steps for following scenario 2

Scenario 2:

  • Job size:
    • Ashley and Bob:
      • Size varies: 3, 6, and 9 node jobs (noted as 3-nodes, 6-nodes and 9-node job)
    • Otto: 3 node jobs
  • There are always jobs waiting, but not always from all users
  • Test cases, submitted to an empty cluster:
    • Ashley submits a 9-node job => the job will run
    • Ashley submits a 6-and 3-node => both jobs will run
    • Ashley submits a 6-node job and Bob a 4-node job => both jobs will run
    • Otto submits four 3 nodes jobs => only 1 job will run, the other 3 jobs are waiting
    • Ashley and Bob submit a 6 node job each => both jobs will run
    • Ashley submits two 3-node jobs, Bob a 4 node job and Otto a 3-node job => all jobs run

regards
deepak

4

Hi Adarsh,
i have created reservation by pbs_rsub and both user can reserve 4-4 compute nodes but if user wanted to submit 6 or 9 node jobs then how to submit jobs with reserve and unreserve node for a single job

user need to submit 9 node job with his 4 reserve node ?

may be some way possible

regards
deepak

5

Another way to look at this is that Ashley can use all the nodes except the 4 set aside for Bob (13 - 4 = 9). Similarly Bob can use all the nodes except the 4 set aside for Ashley (also 9). Everyone else can use all except the 4 + 4 set aside for Ashley and Bob (13 - 8 = 5).

Therefore, set

max_run_res.nodect = [u:PBS_GENERIC=5]
max_run_res.nodect += [u:ashley=9]
max_run_res.nodect += [u:bob=9]
2 Likes
6

Thanks dtalcott

i will test it and get back to you