# Qsub : submit jobs with 17 procs on 5 nodes **URL:** \ **Category:** Users/Site Administrators\ **Created:** [April 1, 2021, 1:18pm UTC](https://community.openpbs.org/t/qsub-submit-jobs-with-17-procs-on-5-nodes/2556 "2021-04-01T13:18:01Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![hagel](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/hagel/32/404_2.png) [@hagel](https://community.openpbs.org/u/hagel)\ **Post date:** [April 1, 2021, 1:18pm UTC](https://community.openpbs.org/t/qsub-submit-jobs-with-17-procs-on-5-nodes/2556/1 "2021-04-01T13:18:01Z")
Hi there, new to openpbs (and this list) I have a problem starting jobs via qsub with i.e. 17 procs on 5 nodes. What I have : A running setup on headnode “i1” with somes nodes in : in12 queue = parallel-19 in17 queue = parallel-19 in20 queue = parallel-19 in22 queue = parallel-19 in46 queue = parallel-19 The default queue is parallel-19 and works fine. I am able to start jobs on these machines. Now I’ m searching the correct way to qsub a job as described in the topic. Testing fast with interactive Jobs I found running i.e. : # qsub -I -l select=5:ncpus=5 -l place=scatter qsub: waiting for job 115.i1 to start qsub: job 115.i1 ready # cat $PBS\_NODEFILE ; echo $NCPUS in12 in17 in20 in22 in46 5 logout qsub: job 115.i1 completed But with this idea starting with more nodes than available is impossible, of course. What I want : Start a job via qsub with 17 procs on the 5 servers available taking the less used servers. Topologie does not matter (in this case). Maybe just a matter of correct syntax or capability of queue ? Any help welcome. ---
**Author:** ![gabe](https://avatars.discourse-cdn.com/v4/letter/g/5daacb/32.png) [@gabe](https://community.openpbs.org/u/gabe)\ **Post date:** [April 1, 2021, 5:41pm UTC](https://community.openpbs.org/t/qsub-submit-jobs-with-17-procs-on-5-nodes/2556/2 "2021-04-01T17:41:07Z")
If you don’t care where the tasks are placed, then you could use: `select=17:ncpus=1` By default, the PBS scheduler should fill up a node before scheduling jobs on the next node (`smp_cluster_dist: pack` in the sched\_config). Gabe ---
**Author:** ![hagel](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/hagel/32/404_2.png) [@hagel](https://community.openpbs.org/u/hagel)\ **Post date:** [April 1, 2021, 8:07pm UTC](https://community.openpbs.org/t/qsub-submit-jobs-with-17-procs-on-5-nodes/2556/3 "2021-04-01T20:07:59Z")
Hi Gabe, thanks for your response. A bit of progress, but unfortunately not complete. Here my result : # qsub -I -l select=17:ncpus=1 qsub: waiting for job 133.i1 to start qsub: job 133.i1 ready udo@in12: /home/udo # cat $PBS\_NODEFILE ; wc -l $PBS\_NODEFILE ; echo $NCPUS in12 in12 in12 in12 in12 in12 in12 in12 in12 in12 in12 in12 in12 in12 in12 in12 in12 17 /var/spool/pbs/aux/133.i1 1 # logout So, i have what I wanted regarding a “good” machinefile : one line for every proc. But : Alle of the nodes are filled by the first node in12, not a “mixture” of nodes. If we talk about file “sched\_config” I see 2 of them : /opt/pbs/etc/pbs\_sched\_config and /var/spool/pbs/sched\_priv/sched\_config which is the one needed ? I suppose “/opt/pbs/etc/pbs\_sched\_config” : # rpm -qf /opt/pbs/etc/pbs\_sched\_config openpbs-server-20.0.1-0.x86\_64 # rpm -qf /var/spool/pbs/sched\_priv/sched\_config file /var/spool/pbs/sched\_priv/sched\_config is not owned by any package (on Centos 8 … ) I changed smp\_cluster\_dist: to “round\_robin” in both if the files, restarting pbs (and the server too), doesn’t change the behaviour. # grep smp\_cluster\_dist /opt/pbs/etc/pbs\_sched\_config /var/spool/pbs/sched\_priv/sched\_config | grep -v “#” /opt/pbs/etc/pbs\_sched\_config:smp\_cluster\_dist: round\_robin /var/spool/pbs/sched\_priv/sched\_config:smp\_cluster\_dist: round\_robin Any further Idea ? Here my config : (ucat is cat without comments) # ucat /opt/pbs/etc/pbs\_sched\_config round\_robin: False all by\_queue: True prime by\_queue: True non\_prime strict\_ordering: false ALL help\_starving\_jobs: true ALL max\_starve: 24:00:00 backfill\_prime: false ALL prime\_exempt\_anytime\_queues: false primetime\_prefix: p\_ nonprimetime\_prefix: np\_ node\_sort\_key: “sort\_priority HIGH” ALL provision\_policy: “aggressive\_provision” resources: “ncpus, mem, arch, host, vnode, aoe, eoe” load\_balancing: true ALL smp\_cluster\_dist: round\_robin fair\_share: false ALL fairshare\_usage\_res: cput fairshare\_entity: euser fairshare\_decay\_time: 24:00:00 fairshare\_decay\_factor: 0.5 preemptive\_sched: true ALL dedicated\_prefix: ded Any further idea ? ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [April 4, 2021, 7:47pm UTC](https://community.openpbs.org/t/qsub-submit-jobs-with-17-procs-on-5-nodes/2556/4 "2021-04-04T19:47:45Z")
Please try any of the below > qsub -l select=17:ncpus=1 -l place=pack -I > Note: last -I after pack is the capital i for Ice Cream, you shoud have a node with atleast 17 cores here > qsub -l select=17:ncpus=1:host=nodename -I > Note: last -I after nodename is the capital i for Ice Cream, you shoud have a node with atleast 17 cores here ---
**Author:** ![hagel](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/hagel/32/404_2.png) [@hagel](https://community.openpbs.org/u/hagel)\ **Post date:** [April 4, 2021, 8:27pm UTC](https://community.openpbs.org/t/qsub-submit-jobs-with-17-procs-on-5-nodes/2556/5 "2021-04-04T20:27:35Z")
Hi adarsh, I tried both of them. Here the results : # qsub -I -l select=17:ncpus=1 -l place=pack -I qsub: waiting for job 135.i1 to start qsub: job 135.i1 ready # cat $PBS\_NODEFILE ; wc -l $PBS\_NODEFILE ; echo $NCPUS in12 in12 in12 in12 in12 in12 in12 in12 in12 in12 in12 in12 in12 in12 in12 in12 in12 17 /var/spool/pbs/aux/135.i1 1 again : 17 procs on in12, no other nodes. # qsub -l select=17:ncpus=1:host=nodename -I^C # won’t start … Aborted with CTL-C # logout qsub: job 135.i1 completed So - no luck. But anway : Thank’s Udo ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [April 4, 2021, 8:53pm UTC](https://community.openpbs.org/t/qsub-submit-jobs-with-17-procs-on-5-nodes/2556/6 "2021-04-04T20:53:56Z")
> [@adarsh](#): > > qsub -l select=17:ncpus=1:host=nodename -I > Note: last -I after nodename is the capital i for Ice Cream, you shoud have a node with atleast 17 cores here Sorry nodename is the hostname of the compute node (pbs\_mom), you can find out by running the below command > pbsnodes -av | grep host ---
**Author:** ![hagel](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/hagel/32/404_2.png) [@hagel](https://community.openpbs.org/u/hagel)\ **Post date:** [April 5, 2021, 8:31am UTC](https://community.openpbs.org/t/qsub-submit-jobs-with-17-procs-on-5-nodes/2556/7 "2021-04-05T08:31:48Z")
Ok, that works, I can select dedicated nodes. But I don’t want to use only one node, but spread my computation over all (in this case 5) nodes and I want the pbs produce a well-adjusted hostfile with the nodes defined in the default queue, i.e. in12 in17 in20 in22 in46 in12 in17 in20 in22 in46 in12 in17 in20 in22 in46 in12 in17 Udo ---
**Author:** ![agurban](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/agurban/32/67_2.png) [@agurban](https://community.openpbs.org/u/agurban)\ **Post date:** [April 5, 2021, 2:32pm UTC](https://community.openpbs.org/t/qsub-submit-jobs-with-17-procs-on-5-nodes/2556/8 "2021-04-05T14:32:57Z")
Take a look at Chapter 5, “Multiprocessor Jobs”, in the PBS Professional User’s Guide. ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [April 5, 2021, 7:04pm UTC](https://community.openpbs.org/t/qsub-submit-jobs-with-17-procs-on-5-nodes/2556/9 "2021-04-05T19:04:27Z")
Please try and adjust the below qsub line based on your requirements > qsub - l select=5:ncpus=10:mpiprocs=10 -l place=scatter ---
**Author:** ![hagel](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/hagel/32/404_2.png) [@hagel](https://community.openpbs.org/u/hagel)\ **Post date:** [April 6, 2021, 3:47pm UTC](https://community.openpbs.org/t/qsub-submit-jobs-with-17-procs-on-5-nodes/2556/10 "2021-04-06T15:47:03Z")
After again lot of time of searching/reading : @ agurban In chapter 5 of 2020.1 user’s Guide (which hopefully corresponden to my installed “openpbs-server-20.0.1-0.x86\_64.rpm” I find a lot of examples (as overall in the net). All good if you have a lot of nodes and you need a symmetric set of processes. But nowhere I see a solution to my problem : Starting odd number of procs with less nodes than my number of processes with load-balances machinefile. Why do I need this ? I have 4 machines dedicated to an mpi-based application which needs i.e. 17 procs. I cannot specify 4 machine always because one may be down. An 17 is a prime number, so no way to compute it “in a matrix”. @adarsh Same here : How realize “prime number” of procs ? Number of really usable nodes “dynamic” . No simple solution ? Ideal : qsub -l ncpus=17 -l place=scatter -I but : qsub: “-lresource=” cannot be used with “select” or “place”, resource is: ncpus ---
**Author:** ![agurban](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/agurban/32/67_2.png) [@agurban](https://community.openpbs.org/u/agurban)\ **Post date:** [April 6, 2021, 4:46pm UTC](https://community.openpbs.org/t/qsub-submit-jobs-with-17-procs-on-5-nodes/2556/11 "2021-04-06T16:46:17Z")
You can specify the number of processes per host, and specify a different host for each chunk: -l select=3:ncpus=3:mpiprocs=3+2:ncpus=4:mpiprocs=4 -lplace=scatter. Note that ncpus is a host-level resource, so you use the select statement, not “-l ncpus”. PBS uses "-l " for job-wide resources such as walltime, not host-level resources. See the User’s Guide, chapter 4, especially 4.3 “Requesting Resources”. Also see especially UG 4.3.3, “Requesting Resources in Chunks”. ---
**Author:** ![agurban](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/agurban/32/67_2.png) [@agurban](https://community.openpbs.org/u/agurban)\ **Post date:** [April 6, 2021, 6:02pm UTC](https://community.openpbs.org/t/qsub-submit-jobs-with-17-procs-on-5-nodes/2556/12 "2021-04-06T18:02:24Z")
> [@hagel](#): > > taking the less used servers You can sort vnodes according to the number of available CPUs: node\_sort\_key: “ncpus HIGH unused” all The default vnode sort is according to priority, which by default is not set. See the Administrator’s Guide, section 4.9.50, “Sorting Vnodes on a Key”. ---
**Author:** ![hagel](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/hagel/32/404_2.png) [@hagel](https://community.openpbs.org/u/hagel)\ **Post date:** [April 6, 2021, 7:12pm UTC](https://community.openpbs.org/t/qsub-submit-jobs-with-17-procs-on-5-nodes/2556/13 "2021-04-06T19:12:04Z")
Ah, finally this works exactly as I wanted : # qsub -I -l select=3:ncpus=3:mpiprocs=3+2:ncpus=4:mpiprocs=4 -lplace=scatter qsub: waiting for job 369.i1 to start qsub: job 369.i1 ready udo@in20: /home/udo # wc -l $PBS\_NODEFILE ; echo $NCPUS 17 /var/spool/pbs/aux/369.i1 3 udo@in20: /home/udo # cat $PBS\_NODEFILE | sort | uniq -c ``` 3 in12 4 in17 3 in20 3 in22 4 in46 ``` I suppose I have to dig deeper into the concepts of vnodes and chunks. The value of $NCPUS is … a bit irritating. Anyway, you may agree, it seem’s to be an config-syntax-nightmare. Missing the old linux KISS-principle. I will play around with these parameters. Hope to be able to find solutions for other unusual configs. So thank you very much. The problem is solved in this case. Have a nice time. Udo ---
**Author:** ![hagel](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/hagel/32/404_2.png) [@hagel](https://community.openpbs.org/u/hagel)\ **Post date:** [April 7, 2021, 9:50am UTC](https://community.openpbs.org/t/qsub-submit-jobs-with-17-procs-on-5-nodes/2556/14 "2021-04-07T09:50:17Z")
So, did some work to procude dynamically the string agurban propsed. Just as a proof of concept. ``` #!/bin/bash # name : generate_qsub_string # proof of concept : produce hopefully well balanced hostfile used by pbs # Will produce string # -l select=...+1... # Input : # p1 : nodes availabe in queue # p2 : cores wanted # Output # string to add to qsub # lets go : ( no checks performed ) # Nodes available NODES_AVAILABLE=$1 # Nodes available minus 1 NODES_AVAILABLE_MINUS_1=$(( $NODES_AVAILABLE - 1 )) # Numer of cores wanted CORES=$2 if [$NODES_AVAILABLE -ge $CORES] then # just one proc per node STRING="-l select=$NODES_AVAILABLE:ncpus=1:mpiprocs=1 -lplace=scatter" echo $STRING "# =>" $NODES_AVAILABLE times 1 core else # do some computing ... AVERAGE_REAL=$( echo "$CORES / $NODES_AVAILABLE" | bc -l ) AVERAGE_INT=$( echo "$CORES / $NODES_AVAILABLE" | bc ) AVERAGE_REST=$( echo $AVERAGE_REAL - $AVERAGE_INT | bc -l ) # ... with a little bit round correction ROUND=$( echo "if ($AVERAGE_REST > 0.5) { print 1 } else { print 0 }" | bc -l ) COUNT1=$(( $AVERAGE_INT + $ROUND )) COUNT2=$(( $CORES - ( $NODES_AVAILABLE_MINUS_1 * $COUNT1 ) )) STRING="-l select=$NODES_AVAILABLE_MINUS_1:ncpus=${COUNT1}:mpiprocs=${COUNT1}+1:ncpus=${COUNT2}:mpiprocs=${COUNT2} -lplace=scatter" echo $STRING "# =>" $NODES_AVAILABLE_MINUS_1 times ${COUNT1} cores, one time ${COUNT2} cores fi ``` Produces : ``` # for available_hosts in {2..6} 18 ; do echo -n $available_hosts hosts " : " ; ./generate_qsub_string $available_hosts 17 ; done 2>&1 2 hosts : -l select=1:ncpus=8:mpiprocs=8+1:ncpus=9:mpiprocs=9 -lplace=scatter # => 1 times 8 cores, one time 9 cores 3 hosts : -l select=2:ncpus=6:mpiprocs=6+1:ncpus=5:mpiprocs=5 -lplace=scatter # => 2 times 6 cores, one time 5 cores 4 hosts : -l select=3:ncpus=4:mpiprocs=4+1:ncpus=5:mpiprocs=5 -lplace=scatter # => 3 times 4 cores, one time 5 cores 5 hosts : -l select=4:ncpus=3:mpiprocs=3+1:ncpus=5:mpiprocs=5 -lplace=scatter # => 4 times 3 cores, one time 5 cores 6 hosts : -l select=5:ncpus=3:mpiprocs=3+1:ncpus=2:mpiprocs=2 -lplace=scatter # => 5 times 3 cores, one time 2 cores 18 hosts : -l select=18:ncpus=1:mpiprocs=1 -lplace=scatter # => 18 times 1 core ``` So, this script is still not very high sophisticated and can be made better, but at a first glance it will be ok for my needs. Thanks again for all help Udo